Title: Continuous Concepts Removal in Text-to-image Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: text-to-image diffusion model

continuous concept removal

responsible AI
Abstract: Text-to-image diffusion models have shown an impressive ability to generate high-quality images from input textual descriptions/prompts. However, concerns have been raised about the potential for these models to create content that infringes on copyrights or depicts disturbing subject matter.
Removing specific concepts from these models is a promising solution to this issue. However, existing methods for concept removal do not work well in practical but challenging scenarios where concepts need to be continuously removed. Specifically, these methods lead to poor alignment between the text prompts and the generated image after the continuous removal process.
To address this issue, we propose a novel concept removal approach called CCRT that includes a designed knowledge distillation paradigm.
CCRT constrains the text-image alignment behavior during the continuous concept removal process by using a set of text prompts.
These prompts are generated through our genetic algorithm, which employs a designed fuzzing strategy. 
To evaluate the effectiveness of CCRT, we conduct extensive experiments involving the removal of various concepts, algorithmic metrics, and human studies.
The results demonstrate that CCRT can effectively remove the targeted concepts from the model in a continuous manner while maintaining the high image generation quality (e.g., text-image alignment).
The code of CCRT is available at https://github.com/wssun/CCRT.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of continuously removing inappropriate or copyrighted concepts from text-to-image (T2I) diffusion models while maintaining text-image alignment. The authors identify that existing concept removal methods fail in practical scenarios requiring sequential removal of multiple concepts. For example, iterative removal of artistic styles (e.g., Van Gogh, Picasso) using state-of-the-art methods like ESD leads to severe \"entity forgetting,\" where the model loses coherence between text prompts and generated images. To solve this, the authors propose CCRT (Continuous Concepts Removal in T2I Diffusion Models), a knowledge distillation framework combined with a genetic algorithm (GA) and fuzzing strategy. Experimental evaluations on Stable Diffusion demonstrate that CCRT outperforms baselines in continuous removal of artistic styles, improper content (e.g., eroticism, violence), and intellectual properties (e.g., Spider-Man). However, the paper is too verbose to contain more substantial experimental results or method details in the main paper. \n\nLast but not least, the authors show obvious hostility towards Russian President Putin. For example, in Figure 10, Putin is used to show eroticism and violence concepts. However, from my perspective, the academic papers should not convey such a view. The same experimental effect can be completely expressed in other forms without using a well-known person.\n\nSTRENGTHS AND WEAKNESSES:\nThe task of continuous concept removal is an interesting task, and it is a practical problem in some real-world scenarios. This work can address real-world needs for iterative model editing in response to evolving copyright or ethical concerns to some extent. Nonetheless, there exists some weaknesses.\n\n1.\tThe main paper refers to the supplementary a lot to describe the proposed method. Which is not suitable. For example, the contents in line 183-198 refer to Figure 7 heavily to describe the necessity of optimized calibration prompt set. \n\n2.\tThe writing of this paper is too verbose. No substantial experimental results are provided even by page 7, which is not suitable for a NeurIPS paper that is limited to 9 pages.\n\nQUESTIONS:\n1.\tThe authors propose to remove semantic concepts continuously one by one. Can some concepts be removed simultaneously? For example, two concepts are removed once. \n\n2.\tThe concepts are removed continuously one by one. It is necessary to show the impact of the removal order of different concepts.\n\n3.\tWhile evaluating the Effectiveness on continuous artistic style removal and Effectiveness on continuous IP removal, the details of training of the classifier for RR-CLS metric should be provided.\n\n4.\tFrom the visualized results in Figure 10 and Figure 11, we can see that the image quality is far from satisfactory when some concepts are removed.\n\nLIMITATIONS:\nThe main experiments are conducted on Stable Diffusion models. However, there are lots of generative models. The proposed method should be applied to other famous models to evaluate the generalization ability of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-07"}

comments: SUMMARY:
Propose a method that can continuously remove unwanted concepts (such as copyrighted artistic styles, sensitive content, or intellectual property) from text-to-image diffusion models while preserving text-image alignment and generation quality. Contributions are:
1. Formulation of the continuous concept removal problem
2. Demonstration that existing methods' drawbacks
3. Methd explanation and illustration of CCRT
4. Experiments and Evaluation

STRENGTHS AND WEAKNESSES:
1. Proposes a novel, well-designed method (distillation + optimized calibration prompts).
2. Strong results: better removal and alignment than prior work.

Weaknesses:
1. Tested mainly on Stable Diffusion; lacks model diversity.
2. Computational cost not reported.
3. No real-world deployment examples
4. Some clarity issues; key details in appendix.

QUESTIONS:
Have you evaluated CCRT on other major text-to-image diffusion models beyond Stable Diffusion v1.4 (e.g., SDXL, DALL-E 3, Midjourney)?
What are the computational costs of CCRT, particularly for the calibration prompt generation (genetic algorithm + fuzzing) and the distillation process?
Could you provide more concrete examples of calibration prompts produced by the genetic algorithm + fuzzing?
How critical is the full genetic + fuzzing calibration prompt search?  Any ablation studies?

LIMITATIONS:
The paper does not address how their method could be repurposed to selectively remove concepts in a way that biases or censors model outputs (e.g., erasing cultural, political, or minority representations unfairly).
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFinding that existing concept removal methods lead to corruption of other concepts (coined “entity forgetting”), this paper introduces CCRT, their own method for sequentially removing undesirable concepts from T2I diffusion models while maintaining alignment for other concepts. CCRT employs a knowledge distillation framework that uses the original model as a teacher. In addition, a genetic algorithm generates optimized calibration prompts for further regularization. The method is tested on artistic styles, NSFW, IP, and objects, showing better-than-SOTA performance in maintaining both concept removal effectiveness and text-image alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Clear Motivation & Problem Identification.\n2. Comprehensive Evals. Multiple concept types (artistic styles, NSFW, IP, objects) are tested, both through quantitative metrics and human studies (tabs 1-4).\n3. Novel Technical Approach. The combination of knowledge distillation with genetic algorithm-generated calibration prompts is creative and interesting. The misalignment distance metric (equation 6 on page 5) provides, in my opinion, a principled way to identify challenging concepts that need further reinforcement.\n\nWeaknesses:\n\n1. Terminology. It is not clear to me why the authors chose to use the term “entity” in “entity forgetting” when they use concepts.\n2. Limited Theoretical Foundation. While the method itself is defined clearly, the paper lacks theoretical analysis of why the proposed approach works and/or under which conditions it might not work. The choice of hyperparameter λ in Equation 5 also seems kind of ad-hoc to me.\n3. Computational Complexity. The genetic algo adds significant computational overhead. Yet, the paper doesn't reflect computational costs of the simpler alternatives.\n4. Limitations in Evals. The human study only has 11 people. No error bars are provided. The comparison is made against ESD while the others (i.e., ICE, MACE, and SPM) are dismissed as weak.\n5. Methodological Concerns. The genetic algo relies heavily on ImageNet hierarchies and LLM-generated synonyms, which may or may not generalize to arbitrary concept removal tasks. Overall, the calibration prompt generation process is complex and may be difficult to reproduce or adapt to new domains.\n6. Limited Scope. The evaluation focuses primarily on Stable Diffusion v1.4. Generalization to more recent models or architectures is not included.\n7. Prior Work. The problems of concept erasure have been identified before, and I would hence suggest the following citations: https://arxiv.org/pdf/2502.13989v1 https://dl.acm.org/doi/10.1145/3635636.3664268 https://arxiv.org/abs/2502.14896\n\nQUESTIONS:\n1. Hyperparam Selection. The paper shows sensitivity to lambda (fig 16) but does not provide a principled method for selecting this — seemingly critical — hyperparameter. How should users choose lambda for new concept removal tasks? ANy theoretical or empirical guidelines?\n2. Beyond Artistic Styles. The bulk of analysis focuses on artistic styles. How well does the genetic algorithm approach work for broader concept groups / abstract ideas etc. I acknowledge there is some limited discussion of this in the appendix but there might be potential to expand this…\n3. Scalability Analysis. The genetic algorithm component appears to be computationally expensive. How does the total computational cost scale with the number of concepts to be removed? \n4. Failure Mode Analysis. The paper would benefit from an analysis of failure cases and guidance on when the method might not be suitable.\n\nLIMITATIONS:\nThe authors adequately address limitations in Section A.6. Some additional points for consideration:\n1. Computational Requirements. More info about computational overhead and when this method might not be suitable.\n2. Domain Generalization. Would love to see a discussion of the limitation of the ImageNet-based genetic algorithm approach for arbitrary concept domains.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method called CCRT to remove concepts from diffusion models, addressing the problem of continuously removing concepts may harm the prompt-image-alignment. CCRT leverages a knowledge distillation framework, guided by a carefully selected calibration prompt set, which is generated through a combination of genetic algorithms and a fuzzing strategy enhanced by large language models (LLMs). The effectiveness of CCRT is validated through both automatic evaluations and human assessments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Practical Motivation: The paper clearly identifies substantial limitations in existing methods when applied to continuous concept removal tasks and proposes a well-reasoned and technically sound solution to address these challenges.\n- Clear Articulation of Failure Cases in Baselines: It provides compelling empirical and visual evidence demonstrating that repeated application of baseline methods leads to prompt misalignment and semantic drift. These failure modes have been underexplored and rarely quantified in prior literature.\n- Well-Structured and Specific Methodological Design: The proposed CCRT (Continuous Concept Removal in Text-to-Image diffusion) framework innovatively integrates concept removal with semantic space preservation via knowledge distillation. This principled approach offers a novel and effective remedy to the semantic drift issue common in continual editing.\n- Comprehensive and Multi-Angled Experimental Evaluation: The paper adopts a holistic evaluation strategy combining diverse automatic metrics with human evaluations to assess both removal quality and semantic alignment. The experimental setup is thorough and methodologically robust.\n\nWeakness\n- In Figure 1, several concepts such as \"Picasso\", \"Monet\", and \"Cezanneo\" are continuously removed after \"Van Gogh\" from the image generated by Stable Diffusion (SD) using the prompt \"defined Van Gogh's work\". However, since the prompt only contains the concept \"Van Gogh\", is it appropriate to use this image to evaluate the removal of other concepts? This choice may raise questions regarding the experimental validity of the continuous removal process.\n- In Figure 6, although the undesirable concept appears successfully removed. However, the generated images no longer exhibit painterly or brush-like visual styles and instead resemble realistic photographs. Despite this clear stylistic discrepancy, these outputs received high scores during the human evaluation process. This discrepancy suggests a potential flaw in either the evaluation criteria or the effectiveness of the calibration mechanism used for prompt alignment.\n- From a technical perspective, the proposed method shares similarities with existing approaches in model editing and concept erasure. However, the experiments are primarily conducted on a set of predefined artistic or sensitive concepts. It is unclear why the authors did not include experiments on more standard datasets such as COCO or CIFAR, which are commonly used in previous works to evaluate concept removal on regular content. Is the proposed method specifically designed for certain types of concepts only?\n- While the paper emphasizes the challenge of \"entity forgetting\" in both the motivation and method sections, the experimental evaluation lacks dedicated analysis to explicitly demonstrate how the proposed method mitigates entity forgetting. A more focused investigation on this aspect would strengthen the claims and clarify the method's effectiveness in preserving non-target concepts.\n\nQUESTIONS:\n1.Limited Quantitative Evidence on Sensitive Content and IP Removal: The paper only provides qualitative visual examples and a limited amount of quantitative evaluations for the “improper content” and “Intellectual Property” categories. There is a lack of comprehensive quantitative analysis in these domains. Without sufficient data, the strength of the conclusions drawn remains uncertain. Could the authors provide more quantitative results for these categories?\n2.Missing Comparisons Against ESD in Table 1: Table 1 omits direct comparisons with ESD, which is a frequently cited strong baseline in this area. Moreover, the authors refer to it as a state-of-the-art method. What is the rationale for excluding ESD from the main benchmark? Additionally, are there scenarios where other baseline methods may outperform the proposed approach in human evaluations? While the paper argues that ESD achieves high scores but poor qualitative results, wouldn't that suggest the need to question the design of the evaluation metrics rather than excluding ESD entirely from the experiments?\n3.Scalability and Computational Efficiency Concerns: The proposed framework seems to require concept-specific fine-tuning for each individual removal task. Could the authors elaborate on the computational efficiency of the approach? In real-world deployment scenarios where hundreds or even thousands of sensitive concepts might need to be removed, how does the method scale? Specifically, how stable and efficient will it remain after the removal of hundreds of concepts?\n4.Effects of Sequential Multi-Step Concept Removal: After removing \"Van Gogh\" followed by \"Picasso\", what are the RR-CLS and RR-LLM scores for the second concept, \"Picasso\"? How does each successive step in the removal process affect the target concept's metrics (e.g., RR-CLS, RR-LLM)? It would be helpful to report per-step effects in sequential removal scenarios.\n5.Theoretical Explanation for Progressive Improvement After Multiple Removals: The paper suggests that removal performance improves over consecutive removal iterations. Do the authors have either a theoretical justification or empirical analysis explaining why multiple removals enhance the model’s effectiveness? Additionally, the concepts “Van Gogh” and “Picasso” are both artists within the same semantic domain. If the two concepts belong to different domains—e.g., \"Van Gogh\" and \"BMW\"—would the same improvement trend still occur?\n6.In Figure 6, although the target concept is successfully removed, the prompt explicitly includes attributes such as “painting,” “brushstrokes,” and “brushwork.” However, the generated images no longer resemble paintings but rather photographs. Does this indicate that the removal of multiple concepts within the same domain affects the global visual style of that domain? After removing four artist-related concepts, is the model still capable of generating simple artworks when prompted with phrases like “a painting”?\n\nLIMITATIONS:\nYes but super simple.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}