Title: Removing Concepts from Text-to-Image Models with Only Negative Samples
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion model

machine unlearning

contrastive learning
Abstract: This work introduces Clipout, a method for removing a target concept in pre-trained text-to-image models. By randomly clipping units from the learned data embedding and using a contrastive objective, models are encouraged to differentiate these clipped embedding vectors. Our goal is to remove private, copyrighted, inaccurate, or harmful concepts from trained models without the need for retraining. This is achieved by considering only negative samples and generating them in a bootstrapping-like manner, requiring minimal prior knowledge. Additionally, theoretical analyses are provided to further understand our proposed Clipout. Extensive experiments on text-to-image show that Clipout is simple yet highly effective and efficient compared with previous state-of-the-art approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce “Clipout” to unlearn private, copyrighted or inappropriate concepts in T2I VLMs, such as Stable Diffusion which include CLIP as a part of the diffusion model. The system is decomposed and unlearning occurs in a single part of the VLM: the encoder that produces the embedding to generate conditions for other parts in order to generate the concept. Clipout randomly “clips out” some units from the embedding and sets them to zero. Other parts remain fixed. The unlearning occurs for a particular concept (concept is described by the prompt in T2I generation) with a novel contrastive objective. The method is introduced with intensive theoretical analyses of the proposed methods.\n\nThe method proves successful in that the concept is erased leaving “related” concepts intact (i.e. can still be generated) and this is achieved with a reduced computational cost compared to other SOTA methods (the latter is demonstrated in the paper).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The authors are commended on presenting a well written paper that has extensive experiments to demonstrate a method of unlearning that can erase target concepts, while ensuring “related” concepts are intact in the generation process\n2. I believe this to be a timely topic to unlearn private, copyrighted, inaccurate and/or harmful content given rising concerns and discourse around these topics\n3. The localisation of the unlearning to the embeddings shows great potential and I am excited for this work to contribute to the related discourse\n4. The implementation is well documented and should be reproducible based on the descriptions\n5. I appreciate the author’s careful reflections on the implementation of these technical fixes “in the real world” (i.e. who decides what should be erased?) that balances the paper\n\n# Weaknesses\nI have outlined some concerns under “Questions” and “Limitations”.\n\n## Minor comments\n1. Please consider including this canonical reference in L27 for “harmful and offensive” content in large, web-scraped datasets: Birhane, Abeba, Vinay Uday Prabhu, and Emmanuel Kahembwe. \"Multimodal datasets: misogyny, pornography, and malignant stereotypes.\" arXiv preprint arXiv:2110.01963 (2021).\n2. The sentence describing what is meant by “concept” in L86-8 is potentially a few words (or should be reworded to make the meaning come across a bit more cleanly)\n\nQUESTIONS:\n1. How are the related concepts chosen and/or identified? Are these intended to be an exhaustive list? What is the potential for unintended ripple effects [1]?\n2. Given that the datasets are known to have limitations, i.e.. Celeb-A is questionably generalisable as it is based on North American celebrities,  and LAION-5b is known for offensive and problematic content that disproportionately impacts marginalised groups [2], how might this impact the generalisability of this work? Under which contexts is this likely to be effective, and are there any where it might not work as well?\n\n**References:**\n\n[1] Amara, Ibtihel, et al. \"EraseBench: Understanding The Ripple Effects of Concept Erasure Techniques.\" arXiv preprint arXiv:2501.09833 (2025).\n[2] Birhane, Abeba, Vinay Uday Prabhu, and Emmanuel Kahembwe. \"Multimodal datasets: misogyny, pornography, and malignant stereotypes.\" arXiv preprint arXiv:2110.01963 (2021).\n\nLIMITATIONS:\nThe authors are commended for reflecting on their limitations. I recognise space is a consideration, but this is an important section and ideally shouldn’t be relegated to the appendix in my opinion. In addition to my questions above, I’d like to ask if the authors have reflected on the following potential biases stemming the use of models for evaluation (i.e. CLIP Score):\n\n1. CLIP is known to have biases [1-4], and given concerns of bias amplification [5], are these results likely to be limited in any way?\n\n**References:**\n\n[1] Hall, Siobhan Mackenzie, et al. \"Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution.\" Advances in Neural Information Processing Systems 36 (2023): 63687-63723.\n\n[2] Hong, Rachel, et al. \"Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp.\" Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 2024.\n\n[3] Agarwal, Sandhini, et al. \"Evaluating clip: towards characterization of broader capabilities and downstream implications.\" arXiv preprint arXiv:2108.02818 (2021).\n\n[4] Berg, Hugo, et al. \"A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.\" arXiv preprint arXiv:2203.11933 (2022).\n\n[5] Hall, Melissa, et al. \"A systematic study of bias amplification.\" arXiv preprint arXiv:2201.11706 (2022).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper proposes a contrastive-based loss to fine-tune the text encoder or concept encoder. By masking some values of the concept embedding and constructing the positive and negative sample pairs, the supervision signals are obtained and used to optimize the concept encoder. Based on such a simple operation, the target concept can be removed from the pretrained diffusion models. The results show some effectiveness of this method with both quantitative and qualitative results.

STRENGTHS AND WEAKNESSES:
Pros:
1) The contrastive unlearning method is quite simple and effective based on a complete theory analysis.
2) This method can take into account both the erasing performance and the memory efficiency, and the baselines compared in experiments are sufficient.

Cons:
The most severe problem is that the whole writing is very poor, making the reading process very difficult. I will give a few examples here:
    a) The abstract part lacks a high-level, intuitive explanation of the contrastive unlearning and indicates which parameters you are optimizing.
    b) In Figure 1, what's the relation between the pretrained encoder in the training stage and the unlearned encoder in the inference stage? The unlearned encoder does not even have any explanations in this figure or the corresponding caption.
    c) In Line 236 ``Personal concept'', this part you are indicating some new application with your technique, but later in Line 246, you start to introduce some implementation details.
These kinds of writing problems are everywhere in this paper, making it hard to read.

QUESTIONS:
What's the intuitive idea of contrastive unlearning?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nClipout introduces a negative-only contrastive unlearning strategy for a text-to-image diffusion model by editing only its text encoder rather than the full generator. Specifically, it takes the text embedding of a target prompt (e.g., “children with guns”) and create randomly masked variants of the embedding.  Then, the encoder is updated with a contrastive loss that pushes the masked variants away each other so that the original concept loses a coherent representation, while leaving other concepts untouched, requiring only these self-generated negative samples.  Theoretical analysis links the objective to maximising alignment loss for the target embedding while preserving the uniformity of the overall feature distribution. From experiments, Clipout achieves larger drops in CLIP similarity and identity-matching scores for the target while reducing impact on unrelated concepts compared to baselines on three scenarios: (1) personalized faces learned via Textual Inversion, DreamBooth, or LoRA; (2) built-in harmful prompts; and (3) copyrighted style prompts such as famous artistic styles.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This work proposes a simple contrastive-learning technique that uses only negative samples and shows that it can erase a target concept without relying on any surrogate concepts.\n\n2. The authors take a novel perspective by analysing concept erasure through the lens of contrastive learning’s two core propeties: alignment for removal of the target concept and uniformity for preserving all the others with theoretical analysis.\n\n**Weaknesses**\n\n1. One of reviewer’s major concerns is the lack of important baselines in this work. For erasing target concepts while preserving the remaining ones, this paper does not compare against several key methods in this field, such as SA [1], UCE [2], MACE [3], SPM [4], and CPE [5]. Many critical components addressed by these prior works for successful unlearning in text-to-image diffusion models are largely ignored. While the proposed method erases concepts without relying on anchor or surrogate concepts, it is essential to compare against these baselines to clearly demonstrate its strengths on performance.\n\n2. Another concern is the lack of comprehensive benchmarks. The experiments are conducted only on object/celebrity erasure with prompts that explicitly contain target concepts. However, recent concept removal approaches emphasize the importance of removing implicitly contained target concepts within prompts, since failure to address this allows for bypassing of the concept erasure. It is unclear whether the proposed prompt-specific method can effectively remove such implicit content. To address this, experiments on I2P benchmakr—a widely used benchmark for evaluating explicit content erasure adopted by many baselines (UCE [2], MACE [3], CPE [5], RECE [6])—are necessary.\n\n3. Recent works on concept removal have identified three essential criteria for successful erasure: efficacy (effectively removing target concepts), specificity/locality (preserving unrelated content), and robustness (resistance to prompt attacks that regenerate erased content). However, this paper does not address robustness, which is a critical issue in this field. Red-teaming tools like RAB [7] and UnlearniDiffAttack [8] have shown that several concept erasure methods are vulnerable to adversarial prompts. Recent methods like CPE, RECE, and AdvUnlearn [9] emphasize the importance of defending against such attacks. The absence of any robustness evaluation in this work is therefore a serious limitation.\n\n4. It also does not examine performance in multi-concept erasure scenarios, as done in UCE, MACE, and CPE. The paper should demonstrate how well the method handles the removal of multiple concepts simultaneously.\n\n5. Despite presenting several theoretical analyses, the paper lacks a theoretical explanation for how preserving uniformity contributes to preserving unrelated concepts. This connection should be theoretically or empirically justified in more rigorous manner.\n\n6. While the authors claim that their fine-tuning process is faster, other efficient baselines have already been proposed after ESD and ASD as training-efficient and even training-free approaches. The paper should include comparisons of computational and memory efficiency with these recent baselines.\n\n**References**\n\n[1] Heng et al. \"Selective amnesia: A continual learning approach to forgetting in deep generative models.\" Advances in Neural Information Processing Systems 36, 2024\n\n[2] Gandikota et al. \"Unified concept editing in diffusion models.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.\n\n[3] Lu et al. \"Mace: Mass concept erasure in diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.\n\n[4] Lyu et al. \"One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.\n\n[5] Lee et al. \"Concept pinpoint eraser for text-to-image diffusion models via residual attention gate.\" The Thirteenth International Conference on Learning Representations, 2025.\n\n[6] Gong et al. \"Reliable and efficient concept erasure of text-to-image diffusion models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[7] Tsai et al. \"Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models?.\" The Twelfth International Conference on Learning Representations, 2024\n\n[8] Zhang et al. \"To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2024.\n\n[9] Zhang et al. \"Defensive unlearning with adversarial training for robust concept erasure in diffusion models.\" Advances in neural information processing systems, 2024.\n\nQUESTIONS:\nQuestions are included in Strengths And Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
The paper introduces Clipout, a unlearning method that removes specific concepts from pre-trained T2I models without retraining. This method randomly clips the embedding of the target concept and uses a contrastive objective to make these altered embeddings dissimilar from each other. The paper validates this method on personal concept removal and built-in concept removal.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper proposes a novel solution for T2I unlearning with only negative samples. 
2. The paper validates across both personal concept removal and built-in concept removal, demonstrating its effectiveness. 
3. The paper shows that its method outperforms other baseline models with high computing efficiency.

Drawback:
The major concern of the paper is that while the paper effectively demonstrates that Clipout preserves the model's ability to generate semantically related concepts, it lacks the evaluation of the model's performance on a broad set of common, unrelated prompts. The current experiments show that the unlearning is precise, but they do not sufficiently prove that it does not degrade the model's core generative quality overall.

QUESTIONS:
1.  The paper proposes to use a random clip of the embedding of the target concept. How about other methods to disrupt representation, like perturbing the target embedding with different levels of Gaussian distribution?
2. Current research shows that the unlearning cannot resist the adversarial attacks. Could the authors analyze the robustness of the proposed method?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03