Continuous Concepts Removal in Text-to-image Diffusion Models
Authors: Tingxu Han
Weisong Sun
Yanrong Hu
Chunrong Fang
Yonglong zhang
Shiqing Ma
Tao Zheng
Zhenyu Chen
Zhenting Wang
Venue: NeurIPS 2025 poster ()
Keywords: text-to-image diffusion model
continuous concept removal
responsible AI
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Text-to-image diffusion models have shown an impressive ability to generate high-quality images from input textual descriptions/prompts. However, concerns have been raised about the potential for these models to create content that infringes on copyrights or depicts disturbing subject matter.
Removing specific concepts from these models is a promising solution to this issue. However, existing methods for concept removal do not work well in practical but challenging scenarios where concepts need to be continuously removed. Specifically, these methods lead to poor alignment between the text prompts and the generated image after the continuous removal process.
To address this issue, we propose a novel concept removal approach called CCRT that includes a designed knowledge distillation paradigm.
CCRT constrains the text-image alignment behavior during the continuous concept removal process by using a set of text prompts.
These prompts are generated through our genetic algorithm, which employs a designed fuzzing strategy. 
To evaluate the effectiveness of CCRT, we conduct extensive experiments involving the removal of various concepts, algorithmic metrics, and human studies.
The results demonstrate that CCRT can effectively remove the targeted concepts from the model in a continuous manner while maintaining the high image generation quality (e.g., text-image alignment).
The code of CCRT is available at https://github.com/wssun/CCRT.

Continuous Concepts Removal in Text-to-image Diffusion Models
Tingxu Han Nanjing University txhan@smail nju. edu, en
Weisong Sun Nanyang Technological University weisong. sunEntu. edu. sg
Yanrong Hu Yangzhou University mx1202405748stu, yzu, edu. en
Chunrong Fangt
Yonglong Zhang
Nanjing University
Yangzhou University y yl zhangggzzz edu. cn
fangchunrongānju edu. cn
Shiqing Ma University of Massachusetts Amherst shiqingma@uuass. edu
Tao Zheng
Zhenyu Chen
Nanjing University zt@nju. edu. cn
Nanjing University zychen@nju. edu, cn
Zhenting Wang Rutgers University zt wwwgg9999mmail.comm
Abstract
Text-to-image diffusion models have shown an impressive ability to generate high- quality images from input textual descriptions/prompis. However, concerns have been raised about the potential for these models to create content that infringes on copyrights depicts disturbing subject matter. Removing specific concepts from these models is a promising solution to this issue. However, existing methods for concept removal do not work well in practical but challenging scenarios where concepts need to be continuously removed. Specifically, these methods lead to poor alignment between the text prompts and the generated image after the continuous removal process. To address this issue, we propose novel concept removal approach called CCRT that includes designed knowledge distillation paradigm. CCRT constrains the text-image alignment behavior during the continuous concept removal process by using a set of text prompts. These prompts are generated through our genetic algorithm, which employs designed fuzzing strategy. To evaluate the effectiveness of CCRT, we conduct extensive experiments involving the removal of various concepts, algorithmic metrics, and human studies. The results demonstrate that CCRT can effectively remove the targeted concepts from the model in continuous manner while maintaining the high image generation quality (e.g., text-image alignment). The code of CCRT is available https: //github. Comm/ssuun/CCR
Introduction
Advancements in Artificial Intelligence Generated Contents (AIGCs) [53] have revolutionized the field of image synthesis [34, 43, 55], among which text-to-image diffusion models enable the creation
of high-quality images from textual descriptions [37, 58). However, this progress has also raised significant concerns regarding the potential misuse of these models [13, 7, 49, 41]. Such misuse
Cooresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

includes generating content that infringes on copyrights, such as mimicking specific artistic styles [35], intellectual properties [51, 52]. or creating disturbing and improper subject matter, including eroticism and violence [38]. Addressing these issues necessitates continuously removing those improper concepts from these models to prevent misuse and protect copyright from infringement.
Existing techniques [10, 18] aiming to remove concepts from the text-to-image diffusion models can be categorized into two types. For given concept that needs to be removed, the first group of methods refines the training data by discarding images containing the undesired concept and then retrains the model from scratch [27, 1, 39]. The other set of methods removes the target concept without requiring full retraining. These methods instead utilize small amount of additional data to fine-tune the models and modify specific neurons [12, 11. 9]. In a real- world scenario, the improper concepts learned by the models, such as copyright-protecte
art styles, are often discovered by the model owner in continuous manner. For example, various artists may continually raise complaints that text-to-image generative AI can replicate their distinctive art style. Additionally, users or red-teaming teams [8] of these models may continuously flag instances where the models generate harmful or malicious content. However, we find that these existing techniques do not perform well in scenarios where different concepts need to be continuously removed one after another, which is practical and important. In detail, we observe that training data filtering methods require model owners to retrain the model from scratch, which is deemed impractical due to its exorbitant cost. The fine-tuning- based methods often struggle maintain the alignment between the text prompts and the generated images after repeated removals, degrading the quality and coherence of the generated content (we discuss such "ennttyy forgetting" problem in Section 3). Thus, it is important to design method that can continuously remove improper concepts learned by text-to-image models with low costs.
In this paper, we propose an approach called CCRT(Continuous Concepts Removal in Text-to-image Diffusion Models) to remove concepts continuously while keeping the text-image alignment of the model. Specifically, we develop knowledge distillation paradigm that concurrently eliminates the unwanted concepts from the model while ensuring the edited model's generation quality and text prompt comprehension ability remain aligned with the original model. This is accomplished by utilizing a collection of prompts produced through our genetic algorithm, which incorporates designed fuzzing strategy. Through extensive experiments, CCRT in removing a variety of concepts continuously. Our results, evaluated using both automated metrics and human studies, show that CCRT can effectively excise targeted concepts such specific artistic styles and improper content while preserving the text-image alignment of the model, ensuring that the output remains faithful to the intended textual descriptions. For example, while keeping continuous concept removal at an average removal rate of 0.87, our method improves the CLIP score from 21.698 25.005 compared to the existing state-of-the-art.
demonstrate the effectiveness of
Our contributions are summarized as follows: I We introduce the continuous concept removal problem, which better represents real-world situations and has more practical applications. 2 We find that existing methods do not work well in the continuous concept removal. In detail, we find that these methods lead to poor alignment between the text prompts and the generated image after the continuous removal process. 3 We propose a novel approach CCRT that can effectively remove concepts continuously while keeping text-image alignment of the text-to-image diffusion models. 4 We conduct a comprehensive evaluation, including automated evaluation and human study. Our experimental results demonstrate CCRT significantly outperforms the state-of-the-art concept removal methods in the continuous concept removal problem.
2 Related Work
T21 diffusion models. Text-to-image (T2I) diffusion models have made significant progress in image synthesis tasks, demonstrating remarkable capabilities in generating high-qualin and diverse images from textual descriptions [33, 22, 56, 31]. One of the most notable open-sourced text-to-image diffusion models is Stable Diffusion [34]. It performs the diffusion process within a latent space derived from a pre-trained autoencoder. The autoencoder reduces the dimensionality of the data samples. Taking this approach allows the diffusion model to leverage the semantic features and visual patterns effectively captured and compressed by the encoder component of the autoencoder.
Concept removal on T2I diffusion models. With the advancements of text-to-image diffusion models, there are also many misuse problems surrounding around them [40, 38, 44, 4]. The generated content of the text-to-image diffusion models can infringe established artistic styles [13] or contain

---

Removed concepts:
Van Gogh
SD
"Van Googl"
- "Picasso"
"Monet"
"Gezannco"
Sunflower
Text prompt: A still life sunflowers, with the bold, post-impressionist style and thick, emotive brushstrokes that defined Gogh's work.
Figure 1: The performance of ESD on removing concepts continuously. It showcases the progress of ESD, continuously removing concepts and the generated images of a fixed text prompt. The leftmost is a true art work of Van Gogh. The right images are generated by Stable Diffusion (SD), ESD (removing "Van Gogh"), ESD (removing "Van Gogh" +"Picasso"), ESD (removing "Van Gogh" *"Picasso" "Monet"), and ESD (removing "Van Gogh" +"Picasso" + "Monet" + "Cezanneo") "), respectively. Observe that the text-image alignment is continuously destroyed as the concept removal process continues, indicating that ESD cannot continuously remove concepts.
improper concepts like pornography and violence [38]. Concept removal a promising way to defend against the misuse problems of diffusion models [20, 11, 12]. In detail, it can make the trained models unlearn the concepts that infringe copyright or contain improper content. The concept removal the text-to-image diffusion models can be view "model editing" process [11, 12, 17, 20, 23, 57, 28, 21] achieved by iinetunnng/moddifiing the model weights. Given the rising of training costs especially on the large-scale models, such lightweight model-editing methods are increasingly sought to alter large-scale generative models with minimal data. These concept removal methods are effective for removing specific concepts learned by the model. However, find that these existing methods do not perform well in the scenario where the concepts need to be continously removed.
3 Motivation
In this section, we introduce the motivation for our approach. We begin by highlighting the importance of continuously removing concepts. We then demonstrate that existing techniques fail to remove the concepts continuously while keeping high generation quality of the model.
3.1 Necessity of continuous concept removal
With the rapid advancement of text-to-image diffusion models, there is increasing need to prevent their potential misuse, such as generating harmful, unethical, or legally infringing content. These risks include generating violent, erotic, sensitive content and replicating copyrighted artistic styles without permission [32, 30]. Removing certain concepts from these models shows promise in addressing this issue. However, model owners/governors often continuously discover improper concepts (c.g., those involving violence or specific artists' copyrighted styles) that the models have learned. For instance, different artists may continuously claim that text-to-image diffusion models like DALL-E 3 [6] and Midjourney [48] can mimic their distinctive styles. Additionally, users may continuously report the generation of malicious content such as violence, guns, and nudity by these models. Thus, model owners/governors require a technique that can swiftly and continuously remove the improper concepts from the deployed models.
3.2 Limitation of existing techniques
A straightforward solution to the issue of continuous concept removal is to reemploy existing tech- niques whenever new concept requires removal. Among them, ESD [1]] is the most representative, which formalizes concept removal into optimization to eliminate the influence of concept æ. However, problem arises during optimization as concepts are not isolated but interconnected with other related concepts. This means that when ESD attempts to eliminate specific concept æ, it causes shift in semantic space of diffusion models. For instance, removing the concepts of artists continuously, such "Van Gogh", "Picasso", "Monet" and "Cezanneo". also affects the concept of "sunflowers"" Such an incidental semantic space shifting becomes more serious as the concept removal continues. Figure illustrates the problem visually. Detailed analysis is shown in Section A.4

---

4 Method
To remove concepts continuously and avoid entity forgetting, we propose CCRT. Our approach relies on the knowledge distillation paradigm, which simultancously removes concepts (removing unwanted knowledge) and aligns the latent semantic space between the original Stable Diffusion models and the edited ones (preserving essential knowledge for text-image alignment). Besides the loss designed for concepts removal, CCRT also incorporates a regularization loss to align the semantic space, whenever a new concept is required to be removed. Additionally, CCRT features entity generation mechanism combining genetic algorithm and fuzzing strategy to generate the searched calibration prompt used in the regularization to enhance effectiveness.
4.1 Distillation for concepts removal and alignment
Problem formulation. The primary objectives of CCRT are removing concepts continuously and keeping the text-image alignment. The removal target can be formulated as:
Eø (zt t) f EB (at, c, t), E
(1)
where Te represents the image x stamped by a noise at timestep t, C the latent concept set to be eliminated, and Eg the diffusion model under concept removal. Intuitively, Equation 1 indicates making (.(-) ) ignore the influence of concept c. On the other hand, the target to keep the text-image alignment can be formulated as:
c9- (2 P. co(21, t), E P\C
where eg. denotes the original diffusion model with frozeen parameters and P\C the input prompt space P that doesn't 't contain concept C. Equation indicates that CCRT should keep the alignment as the stable diffusion model when given text prompts that are irrelevant to the removed concepts.
Continuous concept removal. Given the original diffusion model with frozen parameters cø- (.), we aim to remove concept e on the diffusion model co(-) initialized by co- (-). Following the previous work [1 IJ, we quantify the negative removal guidance direction of c as follows:
De = Eg* (Ite,) t) - 7 [50- (azz, c, t) - eg- (xz, t)]
In particular, the term [cg- (aze, c, t) cg- (Iz, t)] represents the additional impact of concept c on noise prediction. The removal loss is adapted from it and deployed iteratively as follows:
f.m (160 c, t) Aellp
where a denotes the generated images and the step of noise in diffusion process. || ||p is the p norm (p- = 1 in our paper). The Equation 3 aims to guide cạ c, t) to a direction that contains no effect of c. Note that Equation 3 operates iteratively and follows memoryless property, meaning that each iteration builds on the model from the previous step rather than the original diffusion model. This approach enables CCRT to adapt to new requirements as they emerge dynamically.
Text-image aligument. As shown in Section 3, iteratively deploying single Equation results in a serious entity forgetting, disrupting text-image alignment severely. Subsequently, introduce an alignment loss to regulate the model's behavior. With some generated entity-related text prompts (called calibration prompt set), we deploy the alignment regularization loss:
Greg MSE(co (æz, e, t), co- (Iz, e, t)), E
where E and (æve, e, t) denote the calibration prompt set and the noise prediction of an entity text prompt e (e.g.. "a picture of sunflower"), respectively. MSF(()) denotes the mean square error function [3]. Model co- (.) means the original diffusion model with frozen parameters, which is taken as the teacher net. The Greg is designed to regularize ca(),, the student net, to mimic the teacher's behavior, (-) freg enables student net co(.) to approximate teacher net cg- (.)'s entity understanding ability, overcoming "entity forgetting".
Knowledge distillation paradigm. A knowledge distillation paradigm is employed to achieve continuous concept removal and maintain text-image alignment simultaneously. We formulate it into an optimization problem with the definitions of Lym and Gregi
min L Crm + A Freg
(5)

---

where À is the hyper- parameter to balance frm and Creg, A 0. CCRT addresses the task of continuous concept removal with text-image alignment by optimizing & via gradient descent, yielding an ideal edited model, eo(.). During the continuous removal process, assume we want to remove concept C₁ at the removal step i. Given the original diffusion model cg- (.) and the model from previous step c (-), which has removed concepts {cI, C2, ...., c,-1), we obtain (-) by deploying distillation between Eg- (.) and ej- (.) on concept C, through loss L, defined in Equation 5.
Necessity of optimized calibration prompt set. During our practice of Equation 5, we find that a random calibration prompt set causes an unstable text-image alignment. Figure 10 supports the evidence. We deploy Equation 5 with calibration set based on randomly selected entities. The left image is generated by the original diffusion model (Stable Diffusion specifically) of the corresponding text prompt, and the edited models generate the right one. The results exhibit oscillatory and unstable behavior, indicating that existing methods perform well some cases but poorly in others. Specifically, the distillation can maintain text- text-image alignment for some entities but may have misalignment for others. This variability arises because different entities used in distillation impact semantic matching differently. In some specific entities, the misalignment becomes more severe (the right case of Figure 10), and we need to harden such semantics. Entities exhibiting higher misalignment with generated images considered more semantically vulnerable, and are thus given priority in the hardening stage. When the calibration prompt set consists of randomly generated entities, the distillation process aligns only random regions of the semantic space. This incomplete alignment leads to undesired results, as illustrated in Figure 10. To address this, we optimize the calibration prompt
generate entities needing alignment most. Such entities serve as anchors within the semantic space, and the entire space is aligned through these entities.
4.2
Calibration prompt set generation
CCRT with bottom_ k
CCRT with top k
In this section, we introduce how generate the calibration prompt set to improve distillation per- formance. Recent research [42, 45] demonstrates that hard sample mining improves model perfor- mance great extent. Hard sample mining tar- gets to identify the samples that are most useful to a specific task. For text-image alignment, hard samples refer to entities of which the model's se- mantic space is more broken, resulting a more severe misalignment. Inspired by it, we introduce a calibration prompt set generation mechanism to mine hard entities and improve distillation perfor- mance. Following the definition of Lreg in Equa- tion 2, we propose Misalignment Distance (MD) based on norm to measure hardness and identify hard entities:
28
0.20
Seare, RR-CLS
0.15
26-
0.10
CLir-Score,
24
0.05
0.76-
6x10-
0.72
3=10-
0.68
CLIP-Score
RR-CLS
MD
Figure 2: Performance of CCRT with entities having top k and bottom k Misalignment Dis- tance (MD) values. CLIP-Score t. RR-CLS f. Note that a higher MD value is associated with increased CLIP-Score and RSR-CLS.
N
M EB, e.)
1 N
Σ
[Iea(e.) Eg- (e:)llo E
(6)
where E denotes the calibration set, initialized by entities from ImageNet classes [36]. co(-) means the diffusion model to be removed concepts and eØ- (--(-) the original diffusion model. The higher MD indicates the more misalignment, the more important we need to reinforce the corresponding semanties. Note that this step, the calibration set consists of entities without accompanying prompt texts. We sort & by Equation 6 and select the top & entities (k: = 10). To validate the impact difference between entities, we also select the bottom & entities as a control group. Two metrics, RR-CLS and CLIP-Score, are utilized to evaluate the concept removal ability and text-image alignment, respectively. A higher CLIP-Score means a better text-image alignment, while a higher RR-CLS reflects a better concept removal ability. Details of the definition can be found in (Section 5. Figure 2 presents the results, where the red bar is taller than the blue one on CLIP--coree and RR-CLS. indicating that "CCRT with top K" performs better than "CCRT with bottom k". Considering MD, we conclude that entities with higher MD result better distillation performance. To mine such hard entities, heuristic algorithms (genetic algorithm specifically) are considered. The genetic algorithm is well-suited for complex problems such as hard entity mining, as efficiently explores large search spaces and evolves solutions to identify valuable entities [2, 16]. To expand the diversity of found
5

---

Algorithm 1 Genetic Algorithm with Fuzzing
Input: Initialized Entity Set: &, Optimization Direction: MD, Original and Edited Diffusion Models:
Cgr, co. Generation Threshold: G
Output: Calibration set
1: E + E sorted by MD(co- E)
D Rank the initialized entity set by misalignment distance Select top-k hard entities
2: & Top-k(E), E' +- 0. +
3: repeat
4: pars < select pe)sc)_ E' < pars
Sample parents from &
5: for 3, ... Llen (pars) / do par1 t pars[i], parz + par s(i i 1] child + crossover(pari. par2)
6:
7:
D Apply crossover rule to get child D> Enhance with fuzzing- based mutation
8:
child + mastaion meutaion_fuzzingge fuzzing(child)
E. + E' "child, +1 10: end for
9:
11: E Top-k(EUEE)
> Select next- next-generation top-k entities Stop if generation threshold exceeds
12:
until g < G
13: return E
entities, we embed fuzzing strategy enhanced by large language model (LLM), which will generate more diverse entities through specific rules, The lerminologies used are summarized in Table 8.
To further explore potential entities with more hardness, we propose Algorithm 1, featuring a genetic algorithm with a fuzzing strategy enhanced by LLM. We first initialize the calibration set by image classes from ImageNet, with each individual containing one entity to start. An individual means an element of the calibration set, consisting of a list of entities, for example. [ ["post exchange" "]. Algo- rithm I aims optimize the calibration set towards increased environment fitness. The optimization direction of each element is evaluated through Equation 6. The terminologies and their meaning summarized in Table 8. Figure 2 shows that higher MI) values identify entities with greater potential to enhance distillation performance. We first sort the initialized entity set by MD and select the top-k entities (lines 1-2). Then, we randomly select individuals as parents (i.c., the individuals used to generate new ones) from È and assign them to a temporary list, pars, with & updated to include the selected individuals (lines 4-6). To generate new high-quality entities with increased MD, we introduce crossover for optimization (line 7). crossover combines two individuals to create a new one by two specified rules. On the one hand, if entities of the individuals have shared parent, the generated individual will be the parent entity. The semantic hierarchy of ImageNet classes follows previous works [15]. For example, the individual generated from the parent individuals ["post exchange") and ["slop chest") is l"commissary"), reflecting their semantic relation. On the other hand, if there is no semantic relationship between the entities, the generated individual will combine both parent entities. For instance, ["cat", "shark"] is generated from ["cat"] and ["shark"]. However, crossover is limited to identifying entities within the initial ImageNet image classes. To discover high-quality entities with greater MD from broader search space, we introduce strategy called ietrationnfuzzinn to generate additional, similar high-quality entities, where CCRT employs LLM to replace randomly selected entities with synonyms (line 8). The mutation fuutatii__uuzzin can be divided into two stages, mutation and fuzzing. The mutation replaces randomly selected entities with synonyms identified by LLM, specifically GPT-4 in our implementation. For example, the result of mutation(("calt) with "kitty". We then implement a fuzzing strategy generate additional entities based the initial set. The fuzzing leverages I.M to create large batches of data, expanding the calibration set with potentially high-quality entities (detailed in Section A.2). For example, fuzzing([ "coffee mug" "I) might produce [ ["desk lamp", "backpack" " "pencil case "J. More details about crossover and nutation_ffut fuzzngg can be found in Section A.2. We then add these generated entities to E' and repeat the generation iteratively until it reaches a threshold pre-defined by the developer (lines 9-12).
"shark" "I) might be ["kitt", "shark") "J, where "car" is replaced
With generated high-quality entities, CCRT then uses LLM to combine entities into semantically coherent text prompts to craft the final calibration prompt set. For instance, an individual with entities [ ("snowbird", "kiny"] might be combined into the text prompt: "A vibrant snowbird perched next to colorful kitty in a lush tropical setting." The prompt for L.LM can he found in Section A.3. Such text prompts consist of the calibration prompt set used in distillation to ensure text-image alignment.

---

"Van Goghh"
"Picasso", "t
SD
lanuli, Evelved Entities
liety, durk Augmented Entiiies
Canoept Under Remove
Imagee Clacoes (Initaal Brates)
Freiy Selection
Calbrotion Prompt
Calleration Prompt Set
Seau Greug
La
1@
dhos
otl...
Edited T2IDM (Stadenn
Edited
start
Crossover
Mutation Puzzing
Generatoo
T2IDM
Otiginal
Under Remoov
( Eaiiy
Angmentation Eariry
(Teachor Ne1)
Input
(6) Calibration Prongt Set Genentionn
(b) Concept Removal
Ouggui
Figure 3: The overview of CCRT on text-to-image diffusion model (T2IDM), CCRT divides the continuous concept removal task into two stages: (a) calibration prompt set generation and (b) concept removal. In the first stage, CCRT utilizes genetic algorithm to generate prompts as a calibration set. Subsequently, CCRT utilize the calibration set remove concepts with distillation mechanism.
4.3 Procedure of CCRT
Figure 3 illustrates the continuous concept removal procedure of CCRT. It consists of two main components: (a) calibration prompt set generation and (b) concept removal.
Given the original diffusion model and specific concept under removal, CCRT first utilizes the entities from ImageNet (image classes) as the initial set. Then, we employ an elaborate hardness identification function, defined by Equation 6, ensuring the selection of the most promising entities for the next phase. After selection, CCRT uses crossover to evolve the calibration set (() and mutation_fuzee
to expand the calibration set (2). The key intuition behind crossover and mutation mutation_fuzzing is to construct entities with higher MD that can act as better semantic anchors, thereby stabilizing the text-image alignment of the edited models. With entities from crossover and mutation_ fuzzing CCRT selects the most promising candidates according to Equation 6 (3)
CCRT then iteratively applies crossover and mutation fuzzing to these refined entities (4) until it reaches a threshold predefined by the developer. CCRT then feeds the entity set into LLM to weave semantically coherent text prompts for each individual (Q), finally outputting the calibration prompt set. In phase (b), a distillation process is implemented. The original diffusion model serves as the teacher net keep the rext-image alignment of the edited model, while the student net is edited to remove specific concepts such as "Van Gogh". This modification ensures concept removal and text-image alignment simultaneously, which is achieved through the generated calibration prompt set.
Evaluation
We apply our proposed method, called CCRT, to the widely employed diffusion model known as Stable Diffusion (SD v1.4 by default) [34]. Our experimental evaluation comprises two distinct components: an automated evaluation and a user study, in which human participants conduct assessments and judgments. We evaluate CCRT on four different aspects: (1). The effectiveness of CCRT in continuous concept removal, such as artist style, improper content, Intellectual Property (IP), and object. (2). The analysis of text-image alignment of CCRT. (3). The efficiency of CCRT. We also conduct an ablation study of CCRTto analyze the influence of each component.
Due to space limitations, we put the results for text-image alignment, efficiency, and ablation study in Section 1, Section 5.2.1, and Section A.7.1 of the Appendix, respectively.
5.1 Experiment Setup
Metrics. The targets of CCRT can be divided into two aspects: removing concepts continuously and maintaining text-image alignment. To evaluate the effectiveness of concept removal, we propose Removal Rate (RR) for measurement. Technically speaking, RR M/N. N means the total number of generated images with prompts that are crafted around the target concept to be deleted, detailed in Section A.3. For example, prompt "A still life of sunflowers that defined Van Gogh's 's work" and target concept "Van Gogh". M denotes how many images don't contain the target concept among the N images. higher RR indicates a better removal capability. There are three different calculation methods: in-context learning based on L.Mss [24, 54, 19] (RR-LLM), binary classifier training (RR- CL.S), and human evaluation. Specifically, the LLM used in our paper is GPT-4. The details and formalized definition of RR-LLM and RR-CLS are shown in Section A.5.1 of the appendix.
7

---

Table I: Comparison of CCRT and other techniques the effectiveness for continuous artistic style removal. Four famous artistic styles are removed continuously in the order of "Van Gogh". " "Picasso", "Monet", "Cezanne". The comparison with another SOTA ESD [11] is in Table 2. ESD removes concepts by totally destroying the semantic space (serious misalignment) as in Section A. Observe that CCRT achieves 0.753 and 0.874 on RR-CLS and RR-LLM on average, indicating that CCRT succeeds in continuous concept removal. RR-CLS 1, RR-LLM 1.
Removed
SD
UCE
MACE
SPM
CCRT (Ours)
Concept
RR-CLS
RR-LLM
RR-CIS
RR-TIM
RR-CTS
RR-LIM
RR-CTS
RR-LLM
RR-CI.
RR-LIM
"Van Gogh"
0,150
0.014
0.393
0.071
0.471
0.271
0.386
0.286
0.743
0.757
+ "PPccaso""
0,000
0.055
0.124
0.008
0,376
0.104
0.224
0.072
0.712
0.872
"Monet"
0.100
0.353
0.947
+ '"Cezanne"
0.186
0.013
0.241
0.044
0.423
0.077
0.373
0.159
0.818
0.918
Average 0.119
0.061
0.215
0.046
0.406
0.150
0.304
0.154
0.753
0.874
Table 2: Results of the human evaluation. Detailed instructions are provided in Section A.10.Thr The values represent the average rank assigned to each method for given target concept. A higher rank (closer to 1) indicates better performance on the corresponding dimension.
Concept Removal Text image Alignment Other Cuncept Preservation
Image Quality
Target Concept
SD ESD CCRT
SD
ESD
CCRT
SD ESD
CCRT
SD ESD CCRT
"Van Gugh"
3.00 1.59
1.41
1.55 2.27
2.18
1.60 2.45
1.95
1,48 2.57
1,95
"Picasso""
3.00 1.73
1.27
1.42
2.64
1.94
1.48 2.50
2.02
1.62 2,43
1.95
"Monet"
3.00 1.34
1.66
1.63 2:53
1.84
1.35 2.70
1.95
1.50 2.43
2.07
"Cezanne"
2.99 1.31
1,70
1.26 2.96
1.78
1,36 2.66
1.98
1.43 2.32
2.25
To evaluate the text-image alignment, we utilize CLIP-Score [14] and VQA-Score [60] to quantify the level of coherence between the generated images and provided text prompts. The prompt usage indicates better model performance
is detailed in Section A.3. A higher in text-image alignment. The experimental results demonstrate that CCRT can maintain a high text-image alignment when removing concepts continuously.
Human study. user study is also conducted for a comprehensive evaluation. Four dimensions, concept removal ability, text-image alignment, image quality, and other concept preservation, are considered. The details of the human study are in Section A. Our study involved 11 total participants, with an average of 150 responses per participant.
Baselines. Five SOTAs (ESD [11], AdvUn [59], UCE [12), MACE [25], and SPM [26]) are deployed iterativeyy to remove concepts continuously during our evaluation. UCE, MACE, and SPM are "weak" baselines that have a poor removal competence as we compare in Table 1. ESD is the "strongest" concept removal baseline and we compare it comprehensively in human study as Table 2. ESD and another "strong" baseline AdvUn are further analyzed text-image alignment in Section A.
5.2 Effectiveness of CCRT.
Effectiveness on continuous artistic style removal. Table 1 presents a comparison of CCRT and other methods, including SD, UCE, MACE, and SPM, in their effectiveness for removing concepts continuously across four artistic styles: "Van Gogh", "Picasso", "Monet", and "Cezanne". The results evaluated regarding RR-CIS and RR-LLM, where higher scores indicate better effectiveness. CCRT consistently surpasses all other techniques in both RR-CI.S and RR-LL.M. On average, CCRT achieves scores of 0.753 in RR-CLS and 0.874 in RR-LLM, reflecting a significant improvement over SD, with gains of 63% in RR-CLS and 81% % in RR-LLM. Compared to the next best-performing method, MACE, CCRT demonstrates an average improvement of 0.347 in RR-CLS and 0.724 in RR-LLM. Notably, when removing the final concept, "Cezanne". CCRT achieves scores of 0.818 in RR-CLS and 0.918 in RR-LLM, while MACE only reaches 0.423 and 0.077 in RR-CLS and RR-LLM, respectively. Note that UCE [12] can remove several concepts at the same time, which may be an alternative to continuous concept removal. However, as noted Section 3.2, some concepts are harder to remove. For instance, "Van Gogh" is deeply embedded due to extensive training data. UCE can not maintain its removal performance on "Van Gogh". Section A. I, we compare CCRT with another SOTA method, ESD [11]. While ESD enables continuous concept removal, it relies
8

---

Table 3: Results of CCRT continuous improper con- tent removal. RR-CLS is taken as the metric. RR-CLS f.
Table 4: Results of CCRT in the removal of famous intellectual properties (IPs).
Improper Concent
SD
CCRT (Ours)
Intellectual Properties
Remove
RR-LLM
"Eroticisn"
"Violence"
+ "Sell-hamm"
"Froticism"
0.39
0.95
0.99
0.99
Spiddr Man"
0.87
"Violence" 0,51
0.69
0.93
0,95
"Super Mario"
0.94
"Self-harm" 0.47
0.63
0.86
0,97
"Iron Man"
0.96
Table 3: Effects of Sequential Multi-Step Concept Removal. We report the changes in RR-CLS and RR-LLM at each step of the continuous removal process. The FR-CLS/RR-LLE of CCRT after removing corresponding concepts are bold. Observe that CCRT will maintain the per-step concept removed during continuous concept removal.
"Van Gugh"
"Picasso"
"Monel"
Target Concept
RR-CI.
RR-LIM
RR-CIS
RR-LIM
RR-CT.S
RR-LIM
Original SD
0.150
0.014
0.000
0.055
0.140
0.160
"Van Gogh"
0.743
0.757
0.064
0.081
0.150
0.171
"Van Gogh + "Picasso"
0.729
0.789
0.712
0.872
0.132
0.211
"Van Goghh "Picasso" "Monet"
0.771
0.791
0.837
0.740
0.947
on disrupting text-image aligmment, further discussed in Section A.6.1, Figure 7 in the appendix showcases the intuitive visual examples of the comparison between CCRT and all baseline methods.
Effectiveness on continuous improper content removal. Our evaluation includes restricting improper content, such as NSFW (not safe for work) material. We use the I2P dataset [38] as the test set to measure the effectiveness of CCRT in continuously removing such content. The removal process begins with the concept of "eroticism", followed by "violence" and "self-harm". As shown in Table 3, the results indicate that CCRT achieves continuous removal of improper content, progressively increasing effectiveness. To evaluate whether the generated image containing improper content, two widely used classifiers are considered. Specifically. Nudenet [5] classifier is used detect "Eroticism" and Q16 [30] for "Violence" and "Self-harm", Figure 13 in the appendix showcases the intuitive examples.
Effectiveness continuous IP removal. We also evaluate CCRT on continuous protected IP concept removal, such as "Spider Man", "Super Mario", and "Iron Man". Table 4 illustrates the results and Figure 14 showcases the visual evidence. Following [49, 50], employ the prompt set provided these studies and apply RR-LLM to evaluate whether CCRT successfully removes the specified concepts. It is evident that CCRT has the capability to continuously remove concepts related to protected IP concepts throughout the process.
Effectiveness continuous object removal. We also extend CCRT to remove three objects (church, tench, and parachute) continuously. CCRT achieves an RR-CLS at 0.99 and keeps the CLIP-S at 25.62 on average, indicating that CCRT has the capability to continuously remove object concepts while maintaining text-image alignment. Figure 8 showcases the visual examples.
Human evaluation results. Table 2 presents the statistical results of human evaluation. Only "strong" concept removal method ESD [11] considered because other methods (UCE, MACE, SPM) only have relatively weak concept removal effects according to Table 1. Column "Concept Removal" shows user rankings for each method's effectiveness in removing the target concept, with higher rankings (closer to 1) indicating better removal. "Text-Image Alignment" ranks alignment quality between images and prompts. "Other Concept Preservation" reflects the retention of non-tate target concepts, where higher rankings indicate less impact other concepts. "Image Quality" ranks by overall image quality. Note that CCRT demonstrates comparable performance to ESD regarding concept removal. Figure 12 illustrates some intuitive visual examples. However, this is due to ESD's 's disruption of the model's semantic structure, resulting in significant text-image misalignment (column "Text-Image Aligmment") and interference with other concepts (column "Other Concept Preservation"), " Only CCRT successfully balances all four objectives: effective concept removal, maintaining text-image alignment, preserving other concepts, and ensuring high image quality.
Per-step RR-CI S/RR-LLM during the continuous concept removal. We design sequential concept-removal experiment to quantify per-step effects each target's 's RR-CLS and RR-LLM as

---

SLis RK-CLS
CLixsScom
Enlte Ockonies
CLP Score
Tiae (psin))
-0-
20
0381 wIOOOOO AUrAA 0.30
Figure 4: Impact each component. Figure Impact of Figure 6: The efficiency of CCRT.
concepts are continuously removed. For example, after removing "Van Gogh" followed by "Picasso", report the RR-CLS and RR-LLM scores for the second concept, "Picasso". 'able 5 reports the results, Observe that CCRT can maintain the per-step concept removed during continuous removal.
Remove concepts from different domains. To verify domain generality, remove "Van Gogh" + "BMW" instead of another artist. We find that, after removing "Van Gogh", CCRT attains RR-CLS/CLIP-S = 0.757/27.16, and after removing "BMW", it attains 0.856/25.07, indicating that CCRT's behavior is not restricted to semantically similar concepts
CCRT on SD-XL. We extend CCRT to one more recent diffusion model, SD-XL [29]. When removing "Van Gogh" > "Monet" > "Picasso" continuously, CCRT obtains RR-CLS/CLIP-S at 0.749/28.44 40 -> 0.761/27.10 10 0.742/25.30 respectively, confirming CCRT's removal capability.
5.2.1 Efficiency of CCRT
The scenario removing concepts dynamic and urgent, necessitating a swift reaction from the third party involved. It is crucial to ensure continuous and efficient removal of concepts. Each removal is light fine-tune (= GPU-hours). In practice, copyright or NSFW abuse reports arrive one concept time, where CCRT targets. If deployment truly needed to purge hundreds of concepts, retraining a brand-new model would be cheaper than any sequential editor, so that extreme case is outside our target scenario. Figure 6 demonstrates the efficiency of our approach. Observe that our method can continuously remove concepts within a reasonable amount time.
Conclusion
In this paper, we introduce a practical yet challenging problem, namely continuous concept removal, for which existing methods demonstrate limited effectiveness. To solve this problem, we propose method based on our designed knowledge distillation paradigm incorporating a genetic algorithm with fuzzing strategy. We conduct comprehensive evaluations, including automated metrics and human evaluation studies. The results demonstrate that our proposed method is highly effective for continuous concept removal while preserving competitive image generation capability.
Acknowledgements
We thank all human study participants for their valuable contributions. We are also grateful to the reviewers for their insightful suggestions and active engagement during the discussion phase. Finally, we thank the Area Chair for recognizing our work. Their feedback has significantly improved the quality of this paper.
References
OpenAL. DALL-E2 preview risks and limitations. 2
[2] Bushra Alhijawi and Arafat Awajan. Genetic algorithms: Theory, genetic operators, solutions, and applications. Evolutionary Intelligence, 1763):245-1256, 2024.
[3] David M Allen. Mean square error of prediction as criterion for selecting variables. Technometrics. 13(3):699 475, 1971.
[4] Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kei-Wei Chang. How well can text-to image generative models understand ethical natural language interventions? arXiv preprint arXiv:2210.15230 2022.
10

---

[5] P Bedapudi. Nudenet: Neural nets for nudity classification, detection and selective censoring, 2019. 9
[6] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science, 2:3, 2023. 3
[7] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag. Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, pages 5253-5270, 2023.
[8] Zhi-Yi Chin, Chich-Ming Jiang, Ching-Chun Huang. Pin-Ya Yu Chen, and Wei-Chen Chiu. Prompt- ing4debugging: Red-teaming text-to-image diffusion models by finding problematic prompts. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024, 2024. 2
[9] Damai Dai, Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493 8502. Association for Computational Linguisties, 2022.
[10] Masane Fuchi and Tomohiro Takagi. Erasing with precision: Evaluating specific concept erasure from text-to-imase image generative models. arXiv e-prints, pages arXiv 2502, 2025.
[11] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufing Kaufmnan and David Bau, Erasing concepts from diffusion models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 2426-2436, 2023. 2, 4, 8, 9, 25, 27, 30
[12] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúiska, and David Bau, Unified concept editing in diffusion models. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024, Waikoloa, HII USA, January 3-8, 2024, pages 5111-5120, 2024. 2, 3, 8, 26
[13] Avijit Ghosh and Genoveva Fossas. Can there be art without an artist? CoRR, abs/2209.0766 0772209.07667,. 2022. 1, 2
[14] Jack Hessel. Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free free evaluation metric for image captioning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 751 7514-7528, 2021. 8, 27
[15] Mihir Hiyer. Imagenet hierarchy. GitHub repository, 2023, Available imagenet- -rrm wwrddee............ooooo.oo 25
[16] John H Holland. Genetic algorithms. Scientific american, (66661)66677 73, 1992. 5
[17] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lchtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12. 2020, virtual, 2020.
[18] Changhoon Kim and Yanjun Qi. A comprehensive survey on concept erasure text-to-image diffusion models. arXiv e-prints, pages arXiv-2502, 2025.
[19] Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. Outfox: LIm- essay detection through in-context learning with adversarially generated examples. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intellignnce EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 21258-28 21266, 2024. 7,27
[20] Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang. Eli Shechtman, Richard Zhang, and Jun-Yan 7hu. Ablating concepts in text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, ICCV 20233 Paris, France, October 1-6, 2023, pages 22634-22645, 2023. 3
[21] Boheng Li, Yanhao Wei, Yankai Fu, Zhenting Wang. Yiming Li, Jie Zhang, Run Wang, and Tianwei Zhang. Towards reliable verification of unauthorized data usage in personalized text-to-image diffusion models. CoRR, abs/2411..004. 2024. 3
[22] Bowen Li, Xiaojuan Qi. Thomas Lukasiewicz, and Philip Tor. Controllable text-to-image image generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pages 20033-273,, 2019. 2

---

[23] Bingchen Liu, Yizhe Zhu, Kunpeng Song. and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2020.
[24] Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. Grace: Empowering tlm-based software vulnerability detection with graph structure and in in-context learning. J. Syst. Softw, 212: 12031, 2024. 7, 27
[25] Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 6430-6440, 2024. 8
[26] Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue. Jungong Han, and Guiguang Ding. One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 7559-7568, 2024.
[27] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam. Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, ICML, 2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 16784-16804, 2021.
[28] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 10743-10752, 2021.
[29] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. 10
[30] Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation unsafe images and hateful memes from text-to-image models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen. Denmark, November 26-30, 2023, pages 3403-3417, 2023. 3,9
[31] Aditya Ramesh, Mikhail Pavlow, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. shot- text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 8821-883 2021.
[32] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. CoRR, abs/2210..440 2022. 3
[33] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lee. Generative adversarial text to image synthesis. In Proceedings of the 33nd International Conference on Machine Learning, ICML. 2016, New York City, NY, USA, June 19-24, 2016, pages 1060-1069, 2016.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In JEEE/CVF Conference Computer Vision and Patlern Recognition. CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685, 2022. 1, 2.7
[35] Kevin Roose. An ai-generated picture won an art prize. artists are not happy. 2022. 2
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Audrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211 12211 2015. 5
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information P Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. 1
[38] Patrick Schramowski, Manuel Brack, Björ Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 22522-22533 2023, 3.9
12

---

[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-texi models. Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. 2
[40] Riddhi Setty. Ai art generators hit with copyright suit over artists' images. Bloomberg Law. Accessed on February, 1:2023, 2023. 2
[41] Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. De-fake: Detection and attribution of fake images generated by text-to-image generation models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 3418 3432, 2023.
[42] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 761-769, 2016.
[43] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, ICMI, 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 3221 32252, 2023.
[44] Lukas Struppek, Dominik Hintersdorf, and Kristian Kersting. The biased artist: Exploiting cultural biases via homoglyphs in text-guided image generation models. CoRR, (tt/2209.08899) 2022. 2
[45] Martin Takác, Avleen Bijral, Peter Richtárik, and Nati Srebro. Mini-batch primal and dual methods for svms. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pages 1022-1030, 2013. 5
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. 29
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 092888, 2023, 29
[48] Ivan M Tsidylo and Chele Esteve Sena. Artificial intelligence as a methodological innovation the training of future designers: Midjourney tools. Information Technologies and Learning Tools, 97())2003, 2023. 3
[49] Zhenting Wang, Chen Chen, Lingjuan Lyu, Dimitris N Metaxas, and Shiqing Ma. Diagnosis: Detecting unauthorized data usages in text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2023. 1,9
[50] Zhenting Wang, Chen Chen, Vikash Schwag, Minzhou Pan, and Lingjuan Lyu. Evaluating and mitigating ip infringement in visual generative ai. CoRR, abs/2406.04ee 04660, 2024. 9
[51] Zhenting Wang, Chen Chen, Yi Zeng, I Lingjuan Lyu, and Shiqing Ma. Where did I come from? origin attribution of ai-generated images. In Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. 2
[52] Zhenting Wang, Vikash Sehwag, Chen Chen, Lingjuan Lyu, Dimitris N Metaxas, and Shiqing Ma. How to trace latent generative model generated images without artificial watermark? In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024, 2024. 2
[53] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Hong Lin. Ai-generated content (aige): A survey, ACM Comput. Surv., 57(5):
2023.
[54] Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, et al. Unilog: Automatic logging via Ilm and in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pages 14:1 14:1-14:12, 2024. 7, 27
[55] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong Yu Liu, and Ping Luo. Raphael: Text- to-image generation via large mixture of diffusion paths. In Advances in Neural Information Processing Sysiems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. 1
13

---

[56] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Sealing autoregressive models for content-rich text-to-image generation. Trans. Much. Learn. Res., 2022, 2022.
[57] Eric Zhang, Kai Wang. Xingqian Xu. Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget text-to-image diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 Workshops. Seattle, WA, USA. June 17-18, 2024. pages 1755-1764, 2023.
[58] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In JEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 3-324,, 2023,
[59] Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong. Ke Ding, and Sijia Liu. Defensive unlcarning with adversarial training for robust concept crasure in diffusion models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. 27
[60] Jingyao 7hu, Stephanie Tonnesen, Greg I. Bryan, and Mary E Putman. It's breeze: The circumgalactic medium of dwarf galaxy is easy strip. arXiv preprint arXiv:2404.00/29 2024. 8, 28
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Section and and Appendix A.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section A.9.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not have theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Section and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification:
We will release our code upon the acceptance of this work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct Stti:///ggiiieeeeeeeuuuuiiiiii ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The computational cost is too high.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eeerpp/eeeeee/eee/ cc cc//publcc//eee haablic/Ethicstuuddeleess
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics during this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Section A.8 in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators or original owners of assets (e.g.. code, data, models), used paper, properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
19

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
Justification: Please see Appendix A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our study does not have any potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
20

---

Answer: [NA]
Justification:
We don't involve LLMs as any important, original, or non-standard compo-
nents.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / //n/ur/ees.p///ee ereeceee2022 /LLM) for what should or should not be described.
21

---

A Appendix
A.1
Some intuitive visual examples.
Removed concepts:
Removed concepts:
Origiiaal SD
"Man Goglt"
"Plcasso"
"Mase"
"Mas Gogh
"Picassa"
"Monet"
MACE
CCRT
Hereooon TiAllped
Aureoodd TTAllpao
Rersoved Mallipne
J Remored Mihhiggsl
Remowed Aigiiie
Remrnned
X Retrenee
Removed
Th-Mipdd
TeAllpool
Th-kigged
ESD
SPM
Rereoved
A Rarroood Th-llgnno
Rerussed Tecligsed
Remoee
Remosed
Berroood Th-klppdd
X Rerooved
Remeredd
ThAAggedd
Th-ligsel
Th-liane
ToAiggoo
Thokigged
Adv
LICE
Removoa
Ranmel ThAligal
Renesse TiAligse
Reneval
X o /TAAllipe
Removel ThAlgged
ThAlignal
X Th-Alggned
TT-Aligeed
TiAAlgal
still life sunflowers, with hold, pott-impressionis style and thick, emotiie brubsstrokes that defined Van Gogit's work.
prompt:
Figure 7: Performance CCRT and other seven baselines intuitively. "Removed" and "TI-Aligned" denote whether method can remove concept successfully maintain text-image alignment. Observe that in the continuous concept removal process, some "strong" methods, such as ESD and AdvUn(learn) can remove concepts but cannot maintain text-image alignment. Other methods, such as MACE, SPM, and UCE, cannot remove the "Van Gogh" concept successfully. Our method, CCRT, achieves continuous concept removal and maintains text-image alignment. To better compare CCRT with other "strong" baselines (ESD and AdyUn), we present the visual examples in the fourth removal of "Cezanneo" in Figure separately.
Removed concepts
"Church"
+ "Parachute"
"Tench"
SS
CCRT
Figure Visualization results of CCRT on continuous object concept removal. Three object concepts, "Church", "Parachute", and "Tench", are randomly selected. Observe that CCRT continuously removes them successfully.
22

---

Text Prompt
Original SD
CCRT
ESD
AdvUn
landscape painting of Mont Suinte- Victoire. depicting the mountain simplified and geometric forms, similar to Cezanne's series of paintings of the mountain.
sweeping lundscupe of Provence countryside, rendered Van Gogh's characteristic thick, swirling the brushstrokes and vibrant colors.
A depiction of starry night, with the thick, emotive brushstrokes and bold of color. A vibrant sunset over wheat field, with the thick brushstrokes and bold colors characteristic of Van Gogh's style.
Martimved
bustling city street, depicted in swirling, chaotic brushwork that defines Van Gogh's unique perspective.
Removed TI-Aligned
Removed TI-Aligned
Removed TI-Aligned
Removed TI-Aligned
Figure 9: Intuitive visual examples the fourth removal of "Cezanneo" after removing "Van Gogh", "Picasso" and "Monet". We compare CCRT with two other "strong" baselines. Observe that only CCRT achieves both concept removal and Text-image alignment.
SD
Edited Model
Edited Model
SD
Edited Model
Text prompt: A vibrant painting a field of irises in full bloom, with bright colors.
Text prompt: A vivid, swirling painting of a starry night sky over cityscape.
Text prompt: A dynamic painting of group of people dancing festive atmosphere.
Figure 10: Performance of distillation with text prompts on random entities. For each example, the left one is generated by edit models and the right by the original T2I diffusion model (T2IDM). Observe that text-image alignment is terrible in some cases.
Removed
Removed
Removed "Picasso"
Removed
Removed
Removed
"Van Gogh"
"Monet"
"Andrew Ferez" "A. Manzaned"
"Thomas Cole
Original SD
UCE
Removal Failed
Removal Successful
Figure 11: Performance of UCE when removing different concepts. Observe that UCE succeeds in removing the concepts "Andrew Ferez" "A. J. Manzaned". and "Thomas Cole" but fails for "Van Gogh", "Monet" and "Picasso".
23

---

Airplane
Goldfish
Grent white shark
School bus
Church
owl
Sea turtleool
Original SD
CCRT
Figure 12: Intuitive results of CCRT "Other Concept Preservation" against original stable diffusion model. Four different concepts that are not must-be-removed are randomly selected. Observe that
even continuous removal (after removing "Van Gogh", "Picasso", and "Monet"), CCRT still keeps the performance of the original stable diffusion model.
Removed "Eroticism" Putin with a chainsaw and corpse in a concrete bunker.
"Violence"
Putin with a chainsaw and a corpse in concrete bunker.
Original SD
CCRT
Original SD
CCRT
is added by the authors. Both prompts are from the I2P NSFW dataset.
Figure 13: Visualization results of CCRT on sensitive contents. Two NSFW concepts, "Eroticism"
and "Violence", are considered. Observe that CCRT continuously removes them successfully.
Removed"Spider -Man"
+ "Super Mario"
Imagine superhero with acrobatic agility and the ability to cling to walls. Wearing a red and blue suit with web pattems, this character shoots webs from wrist devices and swings across cityscapes, fighting crime with spider-like instincts and an iconic mask that hides his identity.
Generate an image of plump, cheerful plumber with bushy mustache, wearing red cap and overalls. He's jumping joyfully, with a background of clouds and green hills. His outfit features bright blue colors, and he exudes a sense of heroism and adventure.
Original SD
CCRT
Original SD
CCRT
Figure 14: Performance of CCRT to remove concepts on famous IPs. Observe that CCRT has the potential to remove concepts like IPs in continuous process.
24

---

A.2
Crossover rules and mutation operators.
The crossover rules used in Section 4.
Crossover Rule 1. For entities belonging to the same parent entity, the offspring is the parent entity. For example, "commissary" is the parent entity of "post exchange" and "slop chest", then the offspring of crossovent "post exchange", "slop chest") is "commissary".
Crossover Rule 2. For entities without ancestral affiliation, they combined into new individual. For example, the offspring of crossover( "toucan""""a""""" "consolidation") is ["consolidation" "toucan") "].
The hierarchy of ImageNet class is referred to [15].
The mutation operators used in Section 4.
Entity replacement. randomly replaces some entities and generates similar ones as the substitute. For example, the result of mutation nutation_fuzzinn( "consolidation", "toucan "J) might be ("snowbird", "toucan") "J, where "consolidation" is replaced with "snowbird".
Entity augmentation. It randomly generates more semantically diverse entities to augment the entities. For example, the result of mutation_fuezing(( coffee mug "J) might be ["desk lamp", "backpack", "pencil case").
A.3 Data overview
"The style <artist work is: satylee,
Prompt for LLM
"As shown in finsilive images.
Prompt for SD
"Plense determing if the last picture in one of the sartist lists art styles, " answer yes
painting
style answer
A still life of sunflowers, with bold. post- impressionist style and thick, emotive brashstrokes that defined Van Goglis week.
cubist interpretation of still life scene, featuring sharp angles geometric slapes reminiscent of Picasso's early work.
garden with vihrant flowers, captured in the loose brushwork and bold colors definod Monet's impressionist style.
Figure 15: The prompt and data we use our evaluation
Text Prompt: lawyer passionately delivers declamation during retrial, surrounded by the
lush foliage of genus Persea.
Text Prompt: Create an image that captures the essence both maidenliness thresher.
3. Text Prompt: A metal plate holds an espresso maker together with screws.
Figure 16: Examples from the generated calibration prompt set. The red words denote the entities in the prompt.
The prompt set we use to generate images is derived from the artist style dataset provided by ESD [11], with each prompt consisting of specific artist style concepts and visual descriptions.
The prompt we use RR-LLLM "The style of <style> work is: <style>, As shown in the first three images. Please determine if the last picture does remove the style of <style>, Just answer yes
25

---

Table 6: Results of the human evaluation other baselines. Results are reported by mean + std. A higher rank (closer to 1) indicates better performance.
Method
МАCE
UCE
SPM
CCRT
Concept Removal
2.30 + 0.27
3.70 + 0.15
2.80 + 0.20
1.20 0.12
Text-image Alignment
2.90 + 0.26
1.90 + 0.21
2.40 + 0.29
1.80 + 0.18
or no. If the painting has no artistic style, answer null. The quality of some images may be poor. Please do not misjudge."
The prompt we use to wave prompt texts from several entities, "I will give you a list of multiple strings, each deseribing different concept, and ask you to build the most concise text that roughly contains these concepts, which can be used as a prompt to generate an image, but only as long as describes the content of the picture. The list is as follows: <concept_ list>.
A.4 Detailed analysis of the motivation.
We present a comparison between the original work of Sunflowers by "Van Gogh" and images produced by diffusion models edited by ESD when various artistic styles are progressively deleted. The process involves iteratively removing styles, starting with "Van Gogh", and then proceeding to eliminate additional styles, "Picasso". The outcomes displayed in Figure reveal concerning trend where the alignment between the text prompt and the generated image deteriorates with each incremental removal of artistic styles. It is not evidence of how well those later concepts are erased, which is measured later with quantitative metrics such as RR-CLS and RR-LLM.
Initially, when removing the "Van Gogh" style, the image closely corresponds to the text prompt. However, as the removal continues, the images deviate further from the original prompt. When removing the "Picasso" style, the focus shifts towards another concept, leading to images primarily featuring portraits with sunflowers playing a minor role. This progression highlights how the iterative use of ESD results in significant shifts within the semantic space. We define this observation as entity forgetting, where the models are challenged to maintain a coherent understanding of entities such as "sunflowers" over continuous iterations of concept removal. To avoid confusion, we refer to the crased target a concept (c.g.. "Van Gogh") and the prompts to preserve entities (c.g., "sunflower"). Entity forgetting is, thereforre, the loss of alignment for non-target entities after removal.
There are also techniques like UCE [12] aiming remove multiple concepts simultaneously. However, in practical application, we find that their performance fluctuates greatly across different concepts. Figure 11 in Section A.1 showcases some examples. Observe that UCE shows satisfactory perfor- mance on certain concepts such as "Andrew Ferez", " "A. J. Manzaned", "Thomas Cole". However, regarding specific concepts (e.g., "Van Gogh", "Monet", "Picasso"), UCE does not deliver the same effectiveness. Due to extensive and relevant training data associated with concepts like "Van Gogh", such concepts are deeply embedded the model representations and difficult to remove entirely.
A.5 Experiments.
A.5.1 Metrics learning.
To train a concept detection classifier, we use the original stable diffusion(SDD to generate training images and ResNet 50 the architecture. Taking the concept "Van Gogh" an example, 1000 images are generated by the SD given text prompts about "Van Gogh" like "a still life of sunflowers that defined Van Gogh's work." Another 1000 images are generated with prompts which are around other similar concepts like "Picasso", "Alfred Sisley". " All 2000 images are taken to train the "Van Gogh" detection classifier, where 0.8 is the training set and 0.2 the test set. Similarly, a separate classifier will be trained from scratch for any given concept. We utilize the Adam optimizer, set learning rate to le-4, batch size to 32, and epochs to 30. The final model achieves 90.7% top-1 accuracy. For each target concept, we have an independent classifier to compute RR-CLS.
26

---

RR-LLM employs LLM to evaluate the level of alignment between images and given concepts (i.e., artistic style). We require the LLM provide binary classification. The definition of RR-LLM is as follows:
N N Σ (L) = Yes}
RR-LLM
(7)
(ælp, c) means the judgement of æ given elaborate prompt on the removal concept and the indicator function. Specifically, [[.-] returns means the answer of L.M p)c) c) "No", indicating that image æ does not remove the concept with given prompt On the other hand, [{}} returns 1 means concept с does have been removed successfully from image æ. Consequently, a higher RR-LLM indicates better performance.
X includes pair of concept descriptions and images generated by SD models. N means the size the text set.
Figure 15 in Appendix A illustrates the data we utilize in our evaluation. The left column prompts are fed the SD model to generate images. The top prompt denotes the p Equation 7. Following previous work [19, 54, 24], we first show LLM some instances to lead model's 's knowledge, based on which we hope L.LM give its judgment according to context semantics.
RR-CLS involves training binary classifier, denoted as fj. for each removal concept c. When fz predicts positively, it means that does not include C₁ indicating that c; has been removed. RR-CLS is calculated as follows:
RR-CLS
(8)
N
40
CLIP-Score (CCRT)
0.8
Score (ESD)
35
CLIP Score (AdvUn)
CLIP Score
Score
VQA-Score (CCRT)
30
0.5
VQA-Score (ESD)
2
VQA-Score (AdvUn)
25
20
0.2
15
Van Gogh
Picasso
Monet
Cezanne
Continuous removal process
Figure 17: Text-image alignment comparison between CCRT and ESD, measured by CLIP-Score and VQA-Score. Observe that CCRT's text-image alignment is always better than ESD during continuous concept removal process.
A.6 CCRT to remove real-world concepts
We extend CCRT to remove three randomly selected objects from COCO and CIFAR (church->tende pernch-ttench--paahhhteee parachute) continuously. Figure shows the qualitative results. Specifically, CCRT achieves RR-CL /RRCCLS()/CLII-SS))) of 0.98/27.30 church, 2.99/26.46 tench, and 0.99/25.62 on parachute. Observe that CCRT can generalize to remove multiple types of concepts.
A.6.1 Text-image alignment of CCRT
During our evaluation, we find that two "strong" baselines, ESD [11] and AdvUn [59], can achieve competitive result with CCRT for concept removal, whereas other methods perform poorly with no point for comparison. However, ESD and AdvUn cannot maintain satisfying text-image alignment as mentioned in Section 3. Figure 17 illustrates the text-image alignment of ESD, AdvUn, and CCRT across different stages of continuous artistic style removal ("Van Gogh", "Picasso". "Monet", and "Cezanne"). . It presents two key evaluation metrics: CLIP-Score [14] (on the left y-axis in red) and
27

---

Table 7: Impact of removal order. We report RR-CLSÍ and CLIP-St to measure concept removal and text-image alignment, respectively, when removing the same concepts but with different orders.
RR-CLS/CLIP "Van Gogh"
+ *"Picasso" 0.71/25.00
+ *MMonet 0.74/22.70
"Van Gogh" 0.74/27.16
+"Monet" 0.73/25.13
+"Picasso" 0.74/23.11
CCRT 0.74/27.16
RR-CLS/CLIP "Monet"
*"Pccasso" 0.73/25.23
+*Van Gogh"
"Monet"
+aann Gogh"
+"Picasso" 0.73/22.97
CCRT 0.72/27.70
0.74/22.68
0.72/27.70
0.74/24.96
RR-CIS/CLIPS
"Picasso"
+ VaVn Gogh"
+"Monet"
"Picasso"
+"Monet"
+ +*Van Gogh"
CCRT 0.74/27.31
0.73/24.86
0.74/23.17
0.74/27.31
0.73/25.12
0.75/22.99
VQA-Score [60] (on the right y-axis in blue). The solid line denotes the results of CCRT, and the dashed lines denote ESD and AdvUn. As the removal process progresses and more artistic styles are stripped from the images, CCRT demonstrates increasing superiority over ESD in CLIP Score and VQA Score. This highlights CCRT's ability to manage better the challenge of continuously removing multiple concepts while still maintaining strong alignment with both text-based descriptions and visual understanding tasks. In summary, taking ESD an example, while CCRT and ESD perform competitively at the start (CCRT improves ESD by 0.03 in VQA-Score and 2.23 in CLIP-Score), CCRT consistently outperforms ESD the removal process progresses, with larger gains of 0.16 in VQA-Score and 5.01 in CLIP-Score by the end. Figure 9 showcases the visual examples.
A.7 Impact of removal order
To validate the removal order of different concepts, evaluate CCRT with all different removal orders of three concepts, "Van Gogh", "Picasso", and "Monet", including: "Van Gogh" + "Picasso" "Monet", "Van Gogh" "Monet" - "Picasso": "Monet" - "Picasso" -> "Van Gogh", "Monet" "Van Gogh" + "Picasso": "Picasso" + "Van Gogh" - "Monet", "Picasso" - "Monet" + "Van Gogh". Table illustrates the results. Observe that CCRT continuously remove concepts while maintaining text-image alignment across different removal orders.
-
+
A.7.1 Ablation Study
We analyze the impact of each component in CCRT: distillation alignment (Greg) and calibration set generation (CSG), and genetic algorithm with fuzzing (GAF). Two key components of GAF, crossover and mutuation. onwzzzinn Figure 4. Removing distillation alignment reduces the CLIP-Score, indicating a significant disruption in text-image alignment. Without the CSG, the model's performance is hindered, resulting in low RR-CLS. Similarly, in the absence of GAF, the CLIP-Scorc and RR-CLS decrease. Additionally, increasing the hyper parameter excessive alignment on the calibration set negatively affects the semantic space. Observe that the RR- CLS is high when À is 1. This is because the text-image alignment is broken severely, and the model generates totally irrelevant images. CCRT introduces the alignment loss on the calibration set of untouched entities anchor the model's 's semantic space, thereby mitigating entity forgetting. At every removal step. CCRT penalizes any drift in their text-image match, so the semantic space stays fixed and non-target concepts are preserved. We also conduct ablation studies on two key components of GAF, crossover and mutuation, oolooinnuuinnnnn Our evaluation shows that the RSR-CIS downgrades 4 percent points on average.
also taken into consideration. The results are shown in
shown in Figure 5, decreases the CLIP-Score, suggesting
CCRT integrates GA to anchor the semantic space, which needs an initial pool of concept names. We utilize ImageNet classes because they are public and diverse. not because the algorithm depends on the ImageNet hierarchy itself. To validate the generalization of CCRT, we rerun the GA with only randomly generated entities by GPT-4o (e.g., car, sunflower) instead of ImageNet classes. Under progressive removals, "Van Gogh", then + "Picasso", then + +"Monet", CCRT with ImageNet anchor attains 0. + 0.71/25.00 - 0.74/22.70. while with random-entity anchor it attains 0.73/26.99 93 -+ 0. 70/25.22 + 0.72/22.83. Observe that GA with random entities achieves competitive results. It concludes that the calibration prompt generation mechanism generalizes under multiple initial pools of concept names.
28

---

Table 8: The terminology/symbol and the meaning utilize during the method.
Terminology/Symboll
Meaning
entity individual
Image classes such as "post exchange", "slop chest"
list of entities such [ppost exchange "], slsoop chest"), and [ ["toucan", "consolidation")
prompt
A text that is woven through the entities of individual. For example, "A vibrant snowbird perched next to a colorful toucan in a lush tropical setting." is woven through ("snowbird", "toucan" "]
generation
single iteration of the algorithm in which the population is evaluated, selected, and then used to produce a new population.
parent individual
The selected individuals from the current population that will be used to produce new individuals in the next generation.
offering individual
New individuals generated from the parent individuals through genetic operations like crossover and mutation. The original diffusion model with frozen parameters.
Eg- (.)
E-())
The edited diffusion model. The potential concept set.
C
Q1: Please rank the three pictures according the degree of matching between the images and the prompt text, with the highest matching degree ranked 1 and the lowest as 3. The text is: A sunset over beach, with the soft brushstrokes and pastel colors.
Rank:
Figure 18: A simple question from our human evaluation.
A.8 Broader Impact.
In this paper, introduce novel technique for the continuous removal of inappropriate concepts in text-to-image diffusion models. This approach enables the step-by-step continuous elimination of undesired content, ensuring that test-to-image models produce outputs that adhere ethical standards and guidelines. Specifically, we frame concept erasure removal loss under an alignment regularization loss. The optimization yields controlled trade-off between erasure and alignment, forcing the edited model to stay close to the original non-target prompts while erasing the unwanted concept. We believe that our method will play a crucial role in promoting the responsible and ethical development of text-to-image diffusion models. It will help mitigate concerns related to harmful or inappropriate content generation while maintaining high performance and creative flexibility.
constrained optimization, minimizing
A.9 Limitations.
We focus on the continuous concept removal problem in the text-to-image diffusion models. There are other types of models in the AIGC field, such as large language models [46, 47]. Developing continuous concept removal methods for these models will be our future direction.
29

---

version still A.10 Human Evaluation Instruction
We provided each participant in the manual experiment with folder containing the experimental dataset and guidance document. To evaluate concept removal ability, we follow the human evaluation conducted [11]. Participants are presented with set of three authentic artwork images illustrating the target concept for removal, sourced from Google, along with one additional image. The additional image a synthetic image generated using a prompt that includes the target concept, created with Stable Diffusion (SD) or concept removal methods (FSD) and CCRT). Similarly, for other concept preservation, For text-image alignment, each participant is given text prompt paired with the corresponding synthetic images produced by different methods. Participants are then instructed to rank these images according to the alignment between the textual description and visual content. Similarly, for image quality, given set of text prompts paired with the corresponding synthetic images, participants are instructed to rank them based on image quality. Figure 18 illustrates simple example from our human evaluation. Observe that "photo-like" outputs still get "high scores" during human evaluation. The reason is that even without painterly strokes, the photo- matches "sunflower" (and similar prompts) more closely. It tops the alignment ranking. Other images usually contain totally different content from the prompt, as we illustrate in Figure 6. These human ranks line up with our CLIP- or VQA-Score, confirming the evaluation and calibration consistent. The contents of the guidance document are as follows:
Guidance This folder contains four types of subfolders named entity, style, others, and coco, with each type containing folders as evaluation items, totaling 64 folders. Each entity folder contains three images to be evaluated along with the prompt text used to generate these images. Please rank these images based on their relevance to the prompt, with indicating the highest match and the lowest.
Each style folder contains three images to be evaluated and three reference images (named refiuum). Based on the reference images, assess the artistic style (e.g.. Van Gogh, Picasso) of each evaluated image for similarity to the references, ranking from most similar (1) to least similar (3). Each others folder also contains three images to be evaluated and three reference images (named refnum). Using the reference images, assess the similarity of the artistic style (e.g.. Van Gogh, Picasso) of each evaluated image, ranking from (highest similarity) to 3 (lowest similarity). Each coco folder contains three images to be evaluated along with the prompts used to generate these images. Please evaluate the quality of each image, considering both image clarity and prompt relevance, with representing the highest match and the lowest.
30


Reviews:
[{"review_id": "6MlKnkGwrB", "reviewer": "Reviewer_n9GX", "summary": "This paper proposes a method called CCRT to remove concepts from diffusion models, addressing the problem of continuously removing concepts may harm the prompt-image-alignment. CCRT leverages a knowledge distillation framework, guided by a carefully selected calibration prompt set, which is generated through a combination of genetic algorithms and a fuzzing strategy enhanced by large language models (LLMs). The effectiveness of CCRT is validated through both automatic evaluations and human assessments.", "questions": "1.Limited Quantitative Evidence on Sensitive Content and IP Removal: The paper only provides qualitative visual examples and a limited amount of quantitative evaluations for the “improper content” and “Intellectual Property” categories. There is a lack of comprehensive quantitative analysis in these domains. Without sufficient data, the strength of the conclusions drawn remains uncertain. Could the authors provide more quantitative results for these categories?\n2.Missing Comparisons Against ESD in Table 1: Table 1 omits direct comparisons with ESD, which is a frequently cited strong baseline in this area. Moreover, the authors refer to it as a state-of-the-art method. What is the rationale for excluding ESD from the main benchmark? Additionally, are there scenarios where other baseline methods may outperform the proposed approach in human evaluations? While the paper argues that ESD achieves high scores but poor qualitative results, wouldn't that suggest the need to question the design of the evaluation metrics rather than excluding ESD entirely from the experiments?\n3.Scalability and Computational Efficiency Concerns: The proposed framework seems to require concept-specific fine-tuning for each individual removal task. Could the authors elaborate on the computational efficiency of the approach? In real-world deployment scenarios where hundreds or even thousands of sensitive concepts might need to be removed, how does the method scale? Specifically, how stable and efficient will it remain after the removal of hundreds of concepts?\n4.Effects of Sequential Multi-Step Concept Removal: After removing \"Van Gogh\" followed by \"Picasso\", what are the RR-CLS and RR-LLM scores for the second concept, \"Picasso\"? How does each successive step in the removal process affect the target concept's metrics (e.g., RR-CLS, RR-LLM)? It would be helpful to report per-step effects in sequential removal scenarios.\n5.Theoretical Explanation for Progressive Improvement After Multiple Removals: The paper suggests that removal performance improves over consecutive removal iterations. Do the authors have either a theoretical justification or empirical analysis explaining why multiple removals enhance the model’s effectiveness? Additionally, the concepts “Van Gogh” and “Picasso” are both artists within the same semantic domain. If the two concepts belong to different domains—e.g., \"Van Gogh\" and \"BMW\"—would the same improvement trend still occur?\n6.In Figure 6, although the target concept is successfully removed, the prompt explicitly includes attributes such as “painting,” “brushstrokes,” and “brushwork.” However, the generated images no longer resemble paintings but rather photographs. Does this indicate that the removal of multiple concepts within the same domain affects the global visual style of that domain? After removing four artist-related concepts, is the model still capable of generating simple artworks when prompted with phrases like “a painting”?", "limitations": "Yes but super simple.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Practical Motivation: The paper clearly identifies substantial limitations in existing methods when applied to continuous concept removal tasks and proposes a well-reasoned and technically sound solution to address these challenges.\n- Clear Articulation of Failure Cases in Baselines: It provides compelling empirical and visual evidence demonstrating that repeated application of baseline methods leads to prompt misalignment and semantic drift. These failure modes have been underexplored and rarely quantified in prior literature.\n- Well-Structured and Specific Methodological Design: The proposed CCRT (Continuous Concept Removal in Text-to-Image diffusion) framework innovatively integrates concept removal with semantic space preservation via knowledge distillation. This principled approach offers a novel and effective remedy to the semantic drift issue common in continual editing.\n- Comprehensive and Multi-Angled Experimental Evaluation: The paper adopts a holistic evaluation strategy combining diverse automatic metrics with human evaluations to assess both removal quality and semantic alignment. The experimental setup is thorough and methodologically robust.\n\nWeakness\n- In Figure 1, several concepts such as \"Picasso\", \"Monet\", and \"Cezanneo\" are continuously removed after \"Van Gogh\" from the image generated by Stable Diffusion (SD) using the prompt \"defined Van Gogh's work\". However, since the prompt only contains the concept \"Van Gogh\", is it appropriate to use this image to evaluate the removal of other concepts? This choice may raise questions regarding the experimental validity of the continuous removal process.\n- In Figure 6, although the undesirable concept appears successfully removed. However, the generated images no longer exhibit painterly or brush-like visual styles and instead resemble realistic photographs. Despite this clear stylistic discrepancy, these outputs received high scores during the human evaluation process. This discrepancy suggests a potential flaw in either the evaluation criteria or the effectiveness of the calibration mechanism used for prompt alignment.\n- From a technical perspective, the proposed method shares similarities with existing approaches in model editing and concept erasure. However, the experiments are primarily conducted on a set of predefined artistic or sensitive concepts. It is unclear why the authors did not include experiments on more standard datasets such as COCO or CIFAR, which are commonly used in previous works to evaluate concept removal on regular content. Is the proposed method specifically designed for certain types of concepts only?\n- While the paper emphasizes the challenge of \"entity forgetting\" in both the motivation and method sections, the experimental evaluation lacks dedicated analysis to explicitly demonstrate how the proposed method mitigates entity forgetting. A more focused investigation on this aspect would strengthen the claims and clarify the method's effectiveness in preserving non-target concepts.", "was_revised": true, "final_justification": "This paper presents several strengths:\n\n- Practical Motivation: The paper clearly identifies substantial limitations in existing methods when applied to continuous concept removal tasks and proposes a well-reasoned and technically sound solution to address these challenges.\n- Clear Articulation of Failure Cases in Baselines: It provides compelling empirical and visual evidence demonstrating that repeated application of baseline methods leads to prompt misalignment and semantic drift. These failure modes have been underexplored and rarely quantified in prior literature.\n- Well-Structured and Specific Methodological Design: The proposed CCRT framework innovatively integrates concept removal with semantic space preservation via knowledge distillation. This principled approach offers a novel and effective remedy to the semantic drift issue common in continual editing.\n- Comprehensive and Multi-Angled Experimental Evaluation: The paper adopts a holistic evaluation strategy combining diverse automatic metrics with human evaluations to assess both removal quality and semantic alignment. The experimental setup is thorough and methodologically robust.\n\n\nMy main concern is that the method does not seem to effectively produce images with a painting style. Figure 6 on page 15 demonstrates the effect of CCRT on four prompts that clearly ask for \"painting\", \"thick, swirling the brushstrokes\",\"thick brushstrokes\" and \"swirling, chaotic brushwork\". However, the images in the second column clearly appear little painting-like and more photo-like. The baselines may not meet the author's standard on \"Removed\"(Figure 4), however, they managed to generate painting. \nBasic painting style collapse of this work remains a significant weakness.\n\nBesides, using a current government leader as a negative example prompt may not align with academic standards and ethics.", "rebuttal": "**Response to Reviewer n9GX**\n\nThank you very much for your thoughtful questions and recognition of the significance of our work. We genuinely appreciate your support and are encouraged by your positive assessment. We hope the following content will address your concerns.\n\n**Q1:** Why use a \"Van Gogh\"-only prompt to show effects on \"Picasso\", \"Monet\", etc. in Figure 1?\n\n**A1:** Thanks for your comment. Figure 1's sole purpose is to visualize \"alignment drift\": with the prompt fixed to \"Van Gogh,\" we see ESD's text-image alignment degrade after each extra removal step. It is not evidence of how well those later concepts are erased, which is measured later with quantitative metrics such as RR-CLS and RR-LLM. We will add related clarifications in the revised version.\n\n**Q2:** Why do \"photo-like\" outputs still get \"high scores\" during human evaluation?\n\n**A2:** Thanks for your thoughtful comment. In the human evaluation, participants are required to rank the images per prompt on each dimension but not give absolute scores. Rank 1 means the image fits the prompt best. Even without painterly strokes, the photo-like version still matches \"sunflower\" (and similar prompts) more closely, so it tops the alignment ranking. Other images usually contain totally different content from the prompt, as we illustrate in Figure 6. These human ranks line up with our CLIP- or VQA-Score, confirming the evaluation and calibration are consistent. We will add related explanations in the revised version.\n\n**Q3:** Why no COCO / CIFAR tests? Does CCRT work only on artist styles?\n\n**A3:** Thanks for your advice. We start with artist styles and sensitive content because they are the most pressing real-world removal targets (copyright & safety) [1, 2] and are the standard benchmarks in recent erasure papers. CCRT can generalize all concepts [3, 4]. As suggested, we extend CCRT to remove three randomly selected objects from COCO and CIFAR(church, tench, and parachute) continuously. The table below illustrates the quantitative results, and Figure 5 in the appendix shows the qualitative results. Observe that CCRT can generalize to remove multiple types of concepts. We will clarify it in the revised version.\n\n| RR-CLS$\\uparrow$/CLIP-S$\\uparrow$ | Church     | Tench      | Parachute  |\n| --------------------------------- | ---------- | ---------- | ---------- |\n| CCRT                              | 0.98/27.30 | 0.99/26.46 | 0.99/25.62 |\n\n[1]. Avijit Ghosh and Genoveva Fossas. Can there be art without an artist? CoRR, abs/2209.07667, 2022.\n\n[2]. Zhenting Wang, et al. Diagnosis: Detecting unauthorized data usages in text-to-image diffusion models.ICLR 2024.\n\n[3]. Shilin Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR 2024.\n\n[4]. Mengyao Lyu, et al. One-dimensional adapter to rule them all: Concepts, diffusion models, and erasing applications.CVPR 2024.\n\n**Q4:** Why does CCRT mitigate entity forgetting? \n\n**A4:** Thanks for your insightful question. CCRT introduces alignment loss on the calibration set of untouched entities to anchor the model's semantic space, through which CCRT mitigates entity forgetting. At every removal step, CCRT penalizes any drift in their text-image match, so the semantic space stays fixed and non-target concepts are preserved. During the ablation study in Appendix A.4.4, we find that without alignment loss, the CLIP-S of CCRT degrades by 30.3% (from 21.5 to 16.5). We will add related explanations to the revised version.\n\n**Q5:** Limited quantitative evidence on sensitive content and IP removal. \n\n**A5:** Thanks for your advice. Table 3 (improper content) and Table 4 (IP) already give related quantitative results. To strengthen the evidence, we add one more target in each domain during the rebuttal. Below are the results. Observe that CCRT can continuously remove concepts of sensitive content and copyrighted IP. We will add related experimental results in the revised version.\n\n| Eroticism      | +Violence         | + Self-harm    |\n| -------------- | ----------------- | -------------- |\n| 0.95           | 0.93              | 0.99           |\n| **Spider Man** | **+ Super Mario** | **+ Iron Man** |\n| 0.89           | 0.91              | 0.96           |\n\n**Q6:** Why is ESD absent from Table 1? Other baseline for human evaluation.\n\nTable 1 omits direct comparisons with ESD. What is the rationale for excluding ESD from the main benchmark? Additionally, are there scenarios where other baseline methods may outperform the proposed approach in human evaluations? \n\n**A6:** Thanks for your advice. Table 1 lists methods that remove concepts while preserving text-image alignment. ESD \"erases\" mainly by breaking that alignment, as shown in Appendix A.4.2, so including it there would distort the comparison. Instead, we analyse ESD separately in Table 2 through human study.\n\nAs suggested, we compare CCRT with other baselines from Table 1, including MACE, UCE, and SPM. Human-evaluation results are now provided below. Due to the time limit, we only evaluate the ability of concept removal and text-image alignment across 5 new participants on the edited model after removing \"Van Gogh\", \"Picasso\", \"Monet\", and \"Cezanne\". The results are reported with the format of mean $\\pm$ std.  \n\nBelow is the ranking, where a score close to 1 indicates better performance. Best results are bold. Observe that CCRT outperforms all baselines on both Concept Removal and Text-image Alignment. We will add related results in the revised version.\n\n| Method   | Concept Removal | Text-image Alignment |\n| -------- | --------------- | -------------------- |\n| MACE     | 2.30 ± 0.27     | 2.90 ± 0.26          |\n| UCE      | 3.70± 0.15      | 1.90 ± 0.21          |\n| SPM      | 2.80 ± 0.20     | 2.40 ± 0.29          |\n| **CCRT** | **1.20 ± 0.12** | **1.80 ± 0.18**      |\n\n**Q7:** Scalability and computational efficiency concerns.\n\n**A7:** Thanks for your suggestion. Each removal is a light fine-tune (≈ 3 GPU-hours; Appendix A 4.3). In practice, copyright or NSFW abuse reports arrive one concept at a time, where CCRT targets. If a deployment truly needed to purge hundreds of concepts, retraining a brand-new model would be cheaper than any sequential editor, so that extreme case is outside our target scenario. We will add related clarifications in the revised version.\n\n**Q8:** Per-step RR-CLS/RR-LLM during the continuous concept removal.\n\n**A8:** Thanks for your valuable question. As suggested, we have added a per-step table as follows. The value of (row, column) indicates the RR-CLS/RR-LLM of the column concept after removing row concepts. \n\nThe RR-CLS/RR-LLM of CCRT after removing corresponding concepts are bold. Observe that CCRT will maintain the per-step concept removed during continuous concept removal. We will add related results and explanations in the revised version.\n\n| RR-CLS$\\uparrow$/RR-LLM$\\uparrow$ | \"Van Gogh\"      | \"Picasso\"       | \"Monet\"         |\n| :-------------------------------- | :-------------- | :-------------- | --------------- |\n| Original SD                       | 0.150/ 0.014    | 0.000/0.055     | 0.140/0.160     |\n| \"Van Gogh\"                        | **0.743/0.757** | 0.064/0.081     | 0.150/0.171     |\n| \"Van Gogh\"+\"Picasso\"              | **0.729/0.789** | **0.712/0.872** | 0.132/0.211     |\n| \"Van Gogh\"+\"Picasso\"+\"Monet\"      | **0.771/0.791** | **0.773/0.837** | **0.740/0.947** |\n\n**Q9:** Explanation for progressive improvement after multiple removals. Removal across different domains.\n\n**A9:** Thanks for your insightful and thoughtful comment. CCRT fine-tunes the model produced by the previous removal step. While our alignment loss anchors the global semantic space, the tiny shifts that do accumulate make the next target concept easier to suppress—so later removals may look slightly stronger. \n\nTo verify domain generality, we remove \"Van Gogh\" → \"BMW\" instead of another artist. Results are as below, confirming that CCRT's behaviour is not limited to semantically similar concepts. We will add related results and more qualitative images in the revised version.\n\n| RR-LLM$\\uparrow$/CLIP-S$\\uparrow$ | Van Gogh    | BMW         |\n| --------------------------------- | ----------- | ----------- |\n| CCRT                              | 0.757/27.16 | 0.856/25.07 |\n\n**Q10:** Model's ability to produce a painting-style image.\n\n**A10:** Thanks for your comment. CCRT deletes only the chosen style, not the model's painting ability. After removing \"Van Gogh,\" the model still draws brush-stroked paintings in other styles (e.g., Monet). Using the prompt \"a painting of a castle with rich brushwork,\" we generate 15 images from the edited model, which removes four artist-related concepts continuously, and ask GPT-4o to label each as a photo or painting. All 15 are classified as paintings, showing that CCRT preserves general artwork generation. We will add related results and more qualitative images in the revised version."}, {"review_id": "m1pq9EdHDC", "reviewer": "Reviewer_JrSs", "summary": "Finding that existing concept removal methods lead to corruption of other concepts (coined “entity forgetting”), this paper introduces CCRT, their own method for sequentially removing undesirable concepts from T2I diffusion models while maintaining alignment for other concepts. CCRT employs a knowledge distillation framework that uses the original model as a teacher. In addition, a genetic algorithm generates optimized calibration prompts for further regularization. The method is tested on artistic styles, NSFW, IP, and objects, showing better-than-SOTA performance in maintaining both concept removal effectiveness and text-image alignment.", "questions": "1. Hyperparam Selection. The paper shows sensitivity to lambda (fig 16) but does not provide a principled method for selecting this — seemingly critical — hyperparameter. How should users choose lambda for new concept removal tasks? ANy theoretical or empirical guidelines?\n2. Beyond Artistic Styles. The bulk of analysis focuses on artistic styles. How well does the genetic algorithm approach work for broader concept groups / abstract ideas etc. I acknowledge there is some limited discussion of this in the appendix but there might be potential to expand this…\n3. Scalability Analysis. The genetic algorithm component appears to be computationally expensive. How does the total computational cost scale with the number of concepts to be removed? \n4. Failure Mode Analysis. The paper would benefit from an analysis of failure cases and guidance on when the method might not be suitable.", "limitations": "The authors adequately address limitations in Section A.6. Some additional points for consideration:\n1. Computational Requirements. More info about computational overhead and when this method might not be suitable.\n2. Domain Generalization. Would love to see a discussion of the limitation of the ImageNet-based genetic algorithm approach for arbitrary concept domains.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clear Motivation & Problem Identification.\n2. Comprehensive Evals. Multiple concept types (artistic styles, NSFW, IP, objects) are tested, both through quantitative metrics and human studies (tabs 1-4).\n3. Novel Technical Approach. The combination of knowledge distillation with genetic algorithm-generated calibration prompts is creative and interesting. The misalignment distance metric (equation 6 on page 5) provides, in my opinion, a principled way to identify challenging concepts that need further reinforcement.\n\nWeaknesses:\n\n1. Terminology. It is not clear to me why the authors chose to use the term “entity” in “entity forgetting” when they use concepts.\n2. Limited Theoretical Foundation. While the method itself is defined clearly, the paper lacks theoretical analysis of why the proposed approach works and/or under which conditions it might not work. The choice of hyperparameter λ in Equation 5 also seems kind of ad-hoc to me.\n3. Computational Complexity. The genetic algo adds significant computational overhead. Yet, the paper doesn't reflect computational costs of the simpler alternatives.\n4. Limitations in Evals. The human study only has 11 people. No error bars are provided. The comparison is made against ESD while the others (i.e., ICE, MACE, and SPM) are dismissed as weak.\n5. Methodological Concerns. The genetic algo relies heavily on ImageNet hierarchies and LLM-generated synonyms, which may or may not generalize to arbitrary concept removal tasks. Overall, the calibration prompt generation process is complex and may be difficult to reproduce or adapt to new domains.\n6. Limited Scope. The evaluation focuses primarily on Stable Diffusion v1.4. Generalization to more recent models or architectures is not included.\n7. Prior Work. The problems of concept erasure have been identified before, and I would hence suggest the following citations: https://arxiv.org/pdf/2502.13989v1 https://dl.acm.org/doi/10.1145/3635636.3664268 https://arxiv.org/abs/2502.14896", "was_revised": true, "final_justification": "As noted in my comment below, some of my concerns about the authors' language choices remain, in an effort to make the paper as easily graspable as possible. That being said, as long as the authors commit to clearly defining their terms, it's a matter of subjective preference.\n\nThe more tangible points raised in my review were, in my opinion, addressed, and so I think it is only appropriate to move from a 'Borderline Accept' to an 'Accept'.", "rebuttal": "**Response to Reviewer JrSs**\n\nWe sincerely thank the reviewer for the encouraging and insightful feedback. We are especially grateful for your recognition of our paper. We hope the following clarifications and added experiments will address your concerns. \n\n**Q1:** Terminology of \"entity\" and \"concept\".\n\n**A1:** Thanks for your comment. To avoid confusion, we reserve concept for the target we erase (e.g., \"Van Gogh\") and entity for the regular prompts that should remain intact (e.g., \"sunflower\"). \"Entity forgetting,\" therefore, means non-target entities lose alignment after a removal step. We will add related clarifications in the revised version.\n\n**Q2:** Some theoretical foundation of why CCRT works and/or under which conditions it might not work. \n\n**A2:**  Thank you for highlighting this point. We frame concept erasure as a constrained optimization, minimizing removal loss under an alignment regularization loss. The optimization yields a controlled trade-off between erasure and alignment, forcing the edited model to stay close to the original on non-target prompts while erasing the unwanted concept.\n\nIn practice, CCRT works reliably when the calibration prompts span the main semantic space. Without the calibration prompts, CLIP-S from CCRT drops from 23.35 to 18.24. Appendix A.4.4 illustrates the detailed results. We will add related explanations in the revised version.\n\n**Q3:** The selection of λ in Eq. 5.\n\n**A3:** Thanks for your advice. The loss of CCRT has two forces: removal (pushes the model away from the target concept) and alignment (pulls it back to the original semantic manifold). λ balances these two forces: too small →removal fails; too large → alignment fails. We have a sensitivity study in Appendix A.4.4 Figure 16 (λ ∈ {0.001, 0.01, 0.1, 1}),  showing performance is flat in 0.01, which we use by default. We will add related explanations and experiments in the revised version.\n\n**Q4:** Computational Complexity. \n\n**A4:** Thanks for your suggestion. The GA runs offline on CPU only (~3 hours per concept), and distillation is the GPU-heavy step. But the time cost of CCRT is incremental because each new concept reuses previous weights and doesn't need to remove concepts from scratch. Appendix A.4.3 provides the efficiency analysis, and we think it is acceptable for practical blocklists. We will explore further speed-ups in future work.\n\n**Q5:** Human evaluation report with error bars, more people, and other baselines. \n\n**A5:**  Thanks for your advice. We re-calculate the human evaluation results and report them with error bars shown as mean ± standard deviation. \n\nBelow are the results of ESD and CCRT after continuously removing four concepts. The results are reported through ranking (1 means best) and best results are bold.\n\n| Method | Concept Removal     | Text‑image Alignment | Other Concept Preservation | Image Quality       |\n| ------ | ------------------- | -------------------- | -------------------------- | ------------------- |\n| ESD    | **1.31 $\\pm$ 0.49** | 2.96 $\\pm$ 0.19      | 2.66 $\\pm$ 0.68            | 2.32 $\\pm$ 0.71     |\n| CCRT   | 1.70 $\\pm$ 0.46     | **1.78 $\\pm$ 0.48**  | **1.98 $\\pm$ 0.55**        | **2.25 $\\pm$ 0.69** |\n\nEach participant in our human study is required to rank outputs from all methods along each evaluation dimension. On average, each person answer 150 questions, which is enough to reduce individual bias and yield robust conclusions. During the rebuttal, we recruit 5 additional participants. The obtained below results are consistent with those in the main paper. Particularly, ESD’s semantic space breakdown leads to strong Concept Removal but weak Text-Image Alignment, Other Concept Preservation, and Image Quality. By contrast, CCRT delivers the best overall performance across all dimensions. Below are the results of additional human evaluation, higher ranking (close to 1) means better performance. Best results are bold.\n\n| Method | Concept Removal     | Text‑image Alignment | Other Concept Preservation | Image Quality       |\n| ------ | ------------------- | -------------------- | -------------------------- | ------------------- |\n| ESD    | **1.42 $\\pm$ 0.53** | 2.86 $\\pm$ 0.45      | 2.81 $\\pm$ 0.54            | 2.38 $\\pm$ 0.71     |\n| CCRT   | 1.65 $\\pm$ 0.44     | **1.39 $\\pm$ 0.50**  | **1.69 $\\pm$ 0.60**        | **2.21 $\\pm$ 0.66** |\n\nTable 1 demonstrates that all other baselines (UCE, MACE, SPM) are too weak to remove concepts successfully. To demonstrate CCRT’s advantage beyond ESD, during rebuttal, we also run a quick human evaluation against those baselines. Five new participants rank outputs between CCRT and MACE after erasing four concepts (1 is the best). CCRT is closer to 1 for both concept removal and text-image alignment, outperforming all baselines. \n\n| Method   | Concept Removal | Text-image Alignment |\n| -------- | --------------- | -------------------- |\n| MACE     | 2.30 ± 0.27     | 2.90 ± 0.26          |\n| UCE      | 3.70± 0.15      | 1.90 ± 0.21          |\n| SPM      | 2.80 ± 0.20     | 2.40 ± 0.29          |\n| **CCRT** | **1.20 ± 0.12** | **1.80 ± 0.18**      |\n\nWe will add all results and related analysis in the revised version.\n\n**Q6:** Is the GA too ImageNet-specific and hard to reproduce?\n\n**A6:** Thanks for your insightful question. The GA only needs an initial pool of concept names to anchor the semantic space. We utilize ImageNet classes because they are public and diverse, not because the algorithm depends on the ImageNet hierarchy itself. We rerun the GA with only randomly generated entities by GPT-4o (e.g., car, sunflower) instead of ImageNet classes. Below are the results. Observe that GA with random entities achieves competitive results. It concludes that the calibration prompt generation mechanism generalizes under  multiple initial pool of concept names. We will add related results and explanations in the revised version.\n\n| RR-CLS$\\uparrow$/CLIP-S$\\uparrow$ | \"Van Gogh\" | + \"Picasso\" | + \"Monet\"  |\n| --------------------------------- | ---------- | ----------- | ---------- |\n| CCRT[ImageNet]                    | 0.74/27.16 | 0.71/25.00  | 0.74/22.70 |\n| CCRT[Random]                      | 0.73/26.93 | 0.70/25.02  | 0.72/22.83 |\n\n**Q7:** CCRT on more recent models.\n\n**A7:** Thank you for the advice. During the rebuttal, we extend CCRT to one more recent diffusion model, SD-XL [1]. Below are the results. Observe that CCRT on SD-XL achieves higher CLIP-S, reflecting SD-XL's stronger image generation ability than SD v1.4. We will add related results in the revised version.\n\n| RR-CLS$\\uparrow$/CLIP-S$\\uparrow$ | “Van Gogh”  | + “Monet”   | + “Picasso” |\n| --------------------------------- | ----------- | ----------- | ----------- |\n| CCRT[SD v1.4 ]                    | 0.743/27.16 | 0.712/25.00 | 0.740/22.70 |\n| CCRT[SD-XL]                       | 0.749/28.40 | 0.761/27.10 | 0.742/25.30 |\n\n**Q8:** Add more citations.\n\nPrior Work. The problems of concept erasure have been identified before, and I would hence suggest the following citations: \n\n[1].Erasing with Precision: Evaluating Specific Concept Erasure from Text-to-Image Generative Models\n\n[2]. Evaluation of Concept Erasing for Artistic Styles in Diffusion Models\n\n[3]. A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Models\n\n**A8:** Thank you for pointing us to these important works. We will add them to the Related Work section to better introduce the problem of concept erasure by \"Previous works[1, 2, 3] emphasize that concept erasure is important in diffusion models\"."}, {"review_id": "WB6JcwYJJf", "reviewer": "Reviewer_xaBb", "summary": "Propose a method that can continuously remove unwanted concepts (such as copyrighted artistic styles, sensitive content, or intellectual property) from text-to-image diffusion models while preserving text-image alignment and generation quality. Contributions are:\n1. Formulation of the continuous concept removal problem\n2. Demonstration that existing methods' drawbacks\n3. Methd explanation and illustration of CCRT\n4. Experiments and Evaluation", "questions": "Have you evaluated CCRT on other major text-to-image diffusion models beyond Stable Diffusion v1.4 (e.g., SDXL, DALL-E 3, Midjourney)?\nWhat are the computational costs of CCRT, particularly for the calibration prompt generation (genetic algorithm + fuzzing) and the distillation process?\nCould you provide more concrete examples of calibration prompts produced by the genetic algorithm + fuzzing?\nHow critical is the full genetic + fuzzing calibration prompt search?  Any ablation studies?", "limitations": "The paper does not address how their method could be repurposed to selectively remove concepts in a way that biases or censors model outputs (e.g., erasing cultural, political, or minority representations unfairly).", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1. Proposes a novel, well-designed method (distillation + optimized calibration prompts).\n2. Strong results: better removal and alignment than prior work.\n\nWeaknesses:\n1. Tested mainly on Stable Diffusion; lacks model diversity.\n2. Computational cost not reported.\n3. No real-world deployment examples\n4. Some clarity issues; key details in appendix.", "was_revised": true, "final_justification": "The rebuttal adds SD-XL results showing consistent performance beyond SD-1.4, reports clear compute costs, and improves deployment realism with real-world targets plus planned case studies. The authors will move key ablations/efficiency to the main text, provide concrete calibration-prompt examples, and include ablations validating the full genetic+fuzzing search; they also commit to a Responsible Use section. Remaining concerns are modest—possible over-suppression of broader painting styles when erasing specific artists, partial reproducibility of the GA+LLM prompt pipeline (seeds/code release), and stronger statistics for the human study. Overall, these updates address my main reservations. I have updated the score correspondingly.", "rebuttal": "**Response to Reviewer XaBb**\n\nThank you very much for your constructive comments and recognition of the originality of our work. We hope the following results and clarifications can address your concerns. We are happy to conduct more experiments and provide more clarifications if needed.\n\n**Q1:** CCRT on models beyond Stable Diffusion v1.4 (e.g., SD-XL, DALL-E 3, Midjourney)? \n\n**A1:** Thank you for the suggestion. Following previous works [1, 2, 3],  CCRT is white-box and removes concepts continuously through model editing. Because DALL·E 3 and Midjourney are closed-source and not publicly editable, they lie outside the scope of this paper. Following previous techniques [1, 2, 3], we conduct the main evaluation on SD v1.4, which is the most popular open-source diffusion model in both research and practice. \n\nDuring the rebuttal, as suggested, we evaluate CCRT on SD-XL and achieve similar performance. The results are illustrated as follows. Observe that CCRT generalizes across models. We will add related results and explanations in the revised version.\n\n| RR-CLS$\\uparrow$/CLIP-S$\\uparrow$ | “Van Gogh”  | + “Monet”   | + “Picasso” |\n| --------------------------------- | ----------- | ----------- | ----------- |\n| CCRT[SD v1.4 ]                    | 0.743/27.16 | 0.712/25.00 | 0.740/22.70 |\n| CCRT[SD-XL]                       | 0.749/28.40 | 0.761/27.10 | 0.742/25.30 |\n\n[1]. Yimeng Zhang, et al. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. NeuralPS 2024.\n\n[2]. Rohit Gandikota, ea al. Unified concept editing in diffusion models. WACV 2024. \n\n[3]. Shilin Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR 2024.\n\n**Q2:** Report on computational cost.\n\n**A2:** Thanks for this comment. Calibration prompt generation runs offline on CPU and needs about 3 hours per concept, and distillation on one A100 costs roughly three GPU hours per removal. Appendix A.4.3 and Figure 17 provide a detailed analysis. We will move related results to the main paper.\n\n**Q3:** No real-world deployment examples\n\n**A3:** We deploy CCRT on Stable Diffusion, one of the most widely used production models. The removal targets are real artist styles and unsafe content, such as NSFW and copyrighted IP. Before CCRT, these elements still appear. After CCRT, they are removed, and text-image alignment remains close to the original. Figures 6, 10, and 11 in our paper illustrate this effect. In the revision, we will add more full prompt-to-image case study examples to show the real-world deployment live improvement.\n\n**Q4:** Some clarity issues and details in Appendix.\n\n**A4:** Thanks for pointing it out. We will \n\n(1) move some detailed comparisons in the motivation to the Appendix, \n\n(2) and move the experimental results, such as ablation studies and efficiency analysis, to the main paper with the saved space. \n\nOnly long motivation comparisons and extra plots will remain in the Appendix. We will put a one-paragraph overview box at the start of Section 3 to illustrate the organization.\n\n**Q5:** More concrete examples of calibration prompts produced by the genetic algorithm + fuzzing.\n\n**A5:** Thanks for your comment. Figure 13 in the Appendix showcases some examples from the generated calibration prompt set. During the rebuttal, we give three more examples as follows. We will add related examples in the revised version.\n\n| Entity                | Prompt                                                       |\n| --------------------- | ------------------------------------------------------------ |\n| Lhasa, hare           | A serene scene featuring a Lhasa Apso dog sitting gracefully beside a playful hare in a lush green garden. |\n| bow tie, pirate, nail | A pirate wearing a bowtie, holding a nail.                   |\n| beaker, conch         | A beaker filled with liquid next to a conch shell on a wooden table. |\n\n**Q6:** Ablation studies on full genetic + fuzzing calibration prompt search.\n\n**A6:** Thank you for this advice. We have conducted the ablation studies in Appendix A.4.4. Figure 15 reports the results. Omitting the full genetic + fuzzing calibration prompt search keeps concept removal but leads to noticeable drops in CLIP-Score (from 23.35 to 18.24), because the alignment loss is based on the calibration prompt. We will add related results in the revised version.\n\n**Q7:** Could CCRT be used to censor or bias outputs?\n\n**A7:** Thanks for your valuable concern. CCRT removes only the concepts that developers place on a reviewed blacklist. The alignment loss locks every other concept in place, so auditors can check that no unintended drift or bias appears. Our deployments aim to continuously remove concepts such as copyrighted styles and NSFW terms instead of cultural or political groups. \n\nAll technique improvements may be repurposed for abuse. For example, hackers may utilize generative LLMs such as ChatGPT to make bombs[1, 2]. We will add a \"Responsible Use\" paragraph outlining expert review, alignment auditing, and external bias checks before any rollout. Related clarifications and discussions will be added to the revised version. \n\n[1]. Security News This Week: A Creative Trick Makes ChatGPT Spit Out Bomb-Making Instructions. 2024.\n\n[2]. Hacker tricked ChatGPT into providing detailed instructions to make a homemade bomb. 2024."}, {"review_id": "0GRvdv0plE", "reviewer": "Reviewer_XZUm", "summary": "This paper addresses the challenge of continuously removing inappropriate or copyrighted concepts from text-to-image (T2I) diffusion models while maintaining text-image alignment. The authors identify that existing concept removal methods fail in practical scenarios requiring sequential removal of multiple concepts. For example, iterative removal of artistic styles (e.g., Van Gogh, Picasso) using state-of-the-art methods like ESD leads to severe \"entity forgetting,\" where the model loses coherence between text prompts and generated images. To solve this, the authors propose CCRT (Continuous Concepts Removal in T2I Diffusion Models), a knowledge distillation framework combined with a genetic algorithm (GA) and fuzzing strategy. Experimental evaluations on Stable Diffusion demonstrate that CCRT outperforms baselines in continuous removal of artistic styles, improper content (e.g., eroticism, violence), and intellectual properties (e.g., Spider-Man). However, the paper is too verbose to contain more substantial experimental results or method details in the main paper. \n\nLast but not least, the authors show obvious hostility towards Russian President Putin. For example, in Figure 10, Putin is used to show eroticism and violence concepts. However, from my perspective, the academic papers should not convey such a view. The same experimental effect can be completely expressed in other forms without using a well-known person.", "questions": "1.\tThe authors propose to remove semantic concepts continuously one by one. Can some concepts be removed simultaneously? For example, two concepts are removed once. \n\n2.\tThe concepts are removed continuously one by one. It is necessary to show the impact of the removal order of different concepts.\n\n3.\tWhile evaluating the Effectiveness on continuous artistic style removal and Effectiveness on continuous IP removal, the details of training of the classifier for RR-CLS metric should be provided.\n\n4.\tFrom the visualized results in Figure 10 and Figure 11, we can see that the image quality is far from satisfactory when some concepts are removed.", "limitations": "The main experiments are conducted on Stable Diffusion models. However, there are lots of generative models. The proposed method should be applied to other famous models to evaluate the generalization ability of the proposed method.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The task of continuous concept removal is an interesting task, and it is a practical problem in some real-world scenarios. This work can address real-world needs for iterative model editing in response to evolving copyright or ethical concerns to some extent. Nonetheless, there exists some weaknesses.\n\n1.\tThe main paper refers to the supplementary a lot to describe the proposed method. Which is not suitable. For example, the contents in line 183-198 refer to Figure 7 heavily to describe the necessity of optimized calibration prompt set. \n\n2.\tThe writing of this paper is too verbose. No substantial experimental results are provided even by page 7, which is not suitable for a NeurIPS paper that is limited to 9 pages.", "was_revised": true, "final_justification": "Thanks for the authors' response. Most of my concerns are solved. However, the visual quality of the generated images produced by SD-1.4 or other duffusion based methods are still doubtful. Based on these, I decide to raise my rating score from 2 to 3.", "rebuttal": "**Response to Reviewer XZUm**\n\nWe thank the reviewer for acknowledging the novelty of our method and the strength of our experimental results. We hope the following results and clarifications can address your concerns. We are happy to provide more clarifications and results if needed.\n\n**Q1:** Writing and structure issues. Some results are in the Appendix.\n\n**A1:** Thank you for pointing this out. In the revision, we will \n\n(1) move Figure 7 and its supporting discussion from the Appendix to the main text, immediately near lines 183-198. This change makes the calibration-prompt rationale fully self-contained without forcing readers to flip to the supplementary, \n\n(2) move some detailed comparisons in the motivation to the Appendix, \n\n(3) collect lengthy qualitative galleries and calibration-prompt examples into Appendix, \n\n(4) with the saved space, move the experimental results to the main paper.\n\n**Q3:** Delete multiple concepts at once.\n\n**A3:** Thanks for your suggestion. In the real world, the misuse report of improper concepts usually emerges one at a time, which is the scenario our paper and most current concept removal techniques [1-3] target. Deleting multiple concepts at once is still an open-problem. The effectiveness of existing approaches that attempt simultaneous removal (e.g., UCE[1]) is limited, as we report in Figure 8. For example, when removing concepts such as \"Van Gogh\", \"Picasso\", and \"Monet\" all at once, UCE has a poor removal rate of 0.273, 0.132, and  0.231. Removing multiple concepts at once is beyond the scope of this paper, and we will further explore it in future work.\n\n[1]. Yimeng Zhang, et al. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. NeuralPS 2024.\n\n[2]. Rohit Gandikota, ea al. Unified concept editing in diffusion models. WACV 2024. \n\n[3]. Shilin Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR 2024.\n\n[4]. Rohit Gandikota, et al. Unified concept editing in diffusion models. 2024.\n\n**Q4:** The removal order of different concepts.\n\n**A4:** Thanks for your comment. As suggested, we evaluate CCRT with all different removal orders of three concepts, \"Van Gogh\", \"Picasso\", and \"Monet\", including: \n\n \"Van Gogh\" --> \"Picasso\" --> \"Monet\", \"Van Gogh\" --> \"Monet\" --> \"Picasso\";\n\n \"Monet\" --> \"Picasso\" --> \"Van Gogh\", \"Monet\" --> \"Van Gogh\" --> \"Picasso\";\n\n \"Picasso\" --> \"Van Gogh\" --> \"Monet\", \"Picasso\" --> \"Monet\" --> \"Van Gogh\";\n\n The table below illustrates the results. Observe that the removal order does not influence CCRT's performance. CCRT can continuously remove concepts while maintaining text-image alignment across different removal orders.\n\n| RR-CLS$\\uparrow$/CLIP-S$\\uparrow$     | \"Van Gogh\"     | + \"Picasso\"      | + \"Monet\"        |\n| ------------------------------------- | -------------- | ---------------- | ---------------- |\n| CCRT                                  | 0.743/27.16    | 0.712/25.00      | 0.740/22.70      |\n| **RR-CLS$\\uparrow$/CLIP-S$\\uparrow$** | **\"Van Gogh\"** | **+ \"Monet\"**    | **+ \"Picasso\"**  |\n| CCRT                                  | 0.743/27.16    | 0.731/25.13      | 0.742/23.11      |\n| **RR-CLS$\\uparrow$/CLIP-S$\\uparrow$** | **\"Monet\"**    | **+ \"Picasso\"**  | **+ \"Van Gogh\"** |\n| CCRT                                  | 0.720/27.70    | 0.738/25.23      | 0.747/22.68      |\n| **RR-CLS$\\uparrow$/CLIP-S$\\uparrow$** | **\"Monet\"**    | **+ \"Van Gogh\"** | **+ \"Picasso\"**  |\n| CCRT                                  | 0.720/27.70    | 0.741/24.96      | 0.739/22.97      |\n| **RR-CLS$\\uparrow$/CLIP-S$\\uparrow$** | **\"Picasso\"**  | **+ \"Van Gogh\"** | **+ \"Monet\"**    |\n| CCRT                                  | 0.749/27.31    | 0.737/24.86      | 0.741/23.17      |\n| **RR-CLS$\\uparrow$/CLIP-S$\\uparrow$** | **\"Picasso\"**  | **+ \"Monet\"**    | **+ \"Van Gogh\"** |\n| CCRT                                  | 0.749/27.31    | 0.732/25.12      | 0.752/22.99      |\n\n**Q5:** Training details of the classifier used for RR-CLS. \n\n**A5:** Thanks for your comment. RR-CLS utilizes ResNet 50 as the backbone. We train it on 2000 images as the training dataset, 1000 for the target concept, and 1000 from other concepts. Adam optimizer, learning rate 1e-4, batch size 32, 30 epochs. The final model reaches 90.7 percent top-1 accuracy. For each target concept, we have an independent classifier to compute RR-CLS. We will add related settings and results in the revised version.\n\n**Q6:** Image quality in Figure 10 and Figure 11.\n\n**A6:** Thanks for pointing it out. The blur comes from the base model SD-1.4, which often fails on fine details at its 512 × 512 resolution and is known to distort hands and faces. We run the same removal on SD-XL, which produces sharp and faithful images. We will update the images of our experiment in the revised version.\n\n[1]. Prompting Pixels. Comparing Stable Diffusion Models. Medium, 2024. \n\n[2].Dustin Podell et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv:2307.01952, 2023\n\n**Q7:** CCRT on other generative models.\n\n**A7:** In line with previous concept removal techniques [1, 2, 3], we designed CCRT to operate on open-source, publicly available text-to-image diffusion backbones. Stable Diffusion (SD) is by far the most widely adopted model in both research and practice. In contrast, other leading generative models, such as DALL·E 3 and Midjourney, are closed-source and not accessible for model editing. Accordingly, following previous works [1–3], our main experiments use SD v1.4 as the testbed.\n During the rebuttal, we have now also applied CCRT to a more recent diffusion backbone, SD-XL. The additional results (shown below) confirm that CCRT can continuously remove concepts across models.\n\n| RR-CLS/CLIP-S  | “Van Gogh”  | + “Monet”   | + “Picasso” |\n| -------------- | ----------- | ----------- | ----------- |\n| CCRT[SD v1.4 ] | 0.743/27.16 | 0.712/25.00 | 0.740/22.70 |\n| CCRT[SD-XL]    | 0.749/28.40 | 0.761/27.10 | 0.742/25.30 |\n\n[1]. Yimeng Zhang, et al. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. NeuralPS 2024.\n\n[2]. Rohit Gandikota, ea al. Unified concept editing in diffusion models. WACV 2024. \n\n[3]. Shilin Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR 2024."}]

Author rebuttal:

Meta-review: The authors have comprehensively addressed all reviewer concerns through extensive additional experiments and clarifications. Key enhancements include: extending CCRT to SD-XL with consistent performance, providing detailed computational cost analysis, expanding human evaluations with error bars and more participants, demonstrating generalizability to object removal (COCO/CIFAR), adding failure mode and ablation studies, and clarifying ethical use. The rebuttal convincingly resolves issues related to painting-style preservation, terminology, scalability, and evaluation rigor, strengthening the paper’s contribution and practicality.
Decision: Accept (poster)