Removing Concepts from Text-to-Image Models with Only Negative Samples
Authors: Hanwen Liu
Yadong MU
Venue: NeurIPS 2025 poster ()
Keywords: diffusion model
machine unlearning
contrastive learning
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: This work introduces Clipout, a method for removing a target concept in pre-trained text-to-image models. By randomly clipping units from the learned data embedding and using a contrastive objective, models are encouraged to differentiate these clipped embedding vectors. Our goal is to remove private, copyrighted, inaccurate, or harmful concepts from trained models without the need for retraining. This is achieved by considering only negative samples and generating them in a bootstrapping-like manner, requiring minimal prior knowledge. Additionally, theoretical analyses are provided to further understand our proposed Clipout. Extensive experiments on text-to-image show that Clipout is simple yet highly effective and efficient compared with previous state-of-the-art approaches.

Removing Concepts from Text-to-lmage Models with Only Negative Samples
Hanwen Liu, Yadong Mu"
Peking University
hanwenliudmsn com, mydēpku. edu, cn
Abstract
This work introduces Clipout, method for removing a target concept in pre- trained text-to-imag models. By randomly clipping units from the learned data embedding and using a contrastive objective, models are encouraged differentiate these clipped embedding vectors. Our goal is to remove private, copyrighted, inaccurate, or harmful concepts from trained models without the need for retraining. This is achieved by considering only negative samples and generating them in a bootstrapping like manner, requiring minimal prior knowledge. Additionally, theoretical analyses are provided to further understand Extensive experiments on eext-to-image show that Clipout is simple yet highly effective and efficient compared with previous state-of-ther -ftt---r-----ttttt approaches.
proposed Cl Clipout.
Introduction
Modern deep learning research is driven by scale. Powered by tremendous resources in computing and data, large-scale models make it possible to connect the vision and language domains using a single architecture (Radford et al. 2021; Dosovitskiy et al. (02T), i.e., vision-language models (VLMs). Modern text-to-image models (Ramesh al. [202 Yasunaga al. 2023), such as the famed generative model Stable Diffusion (Rombach et al.] (002)). are capable of high high-quality and diverse image synthesis based text prompt. For image-to torgeettt-exttte tasks (Stefanini et al. (023), based on the large-scale data of image-text pairs, VLMs (Li et al.) 2022b) can describe visual concepts of image in natural language. These studies continue evolve and scale, leading various cases in real-world scenarios (Li et al.) 2022a Xu et al. 2025).
While the scaled-up models and datasets have made monumental progress, there is increasing risk that the learned concepts in the trained models may be used maliciously. By leveraging personalized text-to-image techniques (Ruiz et al. 2023) Gal et al., 2023). personal portraits can be forged into deepfakes (Shiohara and Yamasaki [022)] Chen et al,] 2021 Yu et al. (021) by text-to-image models, also be easily imitated without authorization
and aided by these models, the art style of paintings (Liang et al., 2023). Besides, since it is nearly impossible to carefully audit large-scale datasets, models trained these unfiltered data may present biased and harmful behaviors (Birhane al. 2021). These misuses of VI.Ms, especially text-to-image models, will severely violate the rights of privacy and copyright, making the development of the scaling model go against its original intention.
feasible solution to these issues is machine unlearning (Xu et al. 2024), which grants people the right to be forgotten. The primary challenge behind machine unlearning is how to efficiently and precisely unlearn the target concept from trained models. Early research work (Bourtoule et al., T02T
Yan et al. 2022) focused on partitioning the dataset and model into several shards and sub-models, and re- trained the sub-mot sub-model corresponding to the target concept. However, these methods require re-den of the model architecture, and cannot be applied modern scaled-up dision-languae
"Correspondngg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

models. Recent efforts have been made toward unlearning copyright or harmful data in trained text- to-image models (Gandikota al., 2023 Kumari et al., 2023). These methods operate by fine-tuning the trained diffusion model in an end-to-end fashion, and there is a trade-off between efficiency, effectiveness, and the side effects introduced by the change of model parameters. As modern deep neural networks, though large in scale, are built upon different foundational artifacts (Radford et al. 202I). this paper aims to investigate the question: Can we remove particular concepts in trained lext-to-image VLMs, by only unlearning the foundational artifacts that build up these models?
To unlearn text-to-image models at scale, this paper introduces a conceptually simple method, Clipout, by focusing on the embedding of the learned concepts. Unlike previous methods that treat the vision-lande languag model as a whole, Clipout decouples the complex -oxttt system into several distinct artifacts (e.g., the CLIP encoder (Radford et al. (02I]) used in Stable Diffusion (Rombach et al. 2. and only unlearns the artifact that responsible for generating the concept embedding. Inspired by the high efficiency and strong generalization of contrastive learning, in this paper, we explore the opposite direction and propose contrastive unlearning, which aims at unlearning particular concept by introducing novel contrastive objective. Based on the contrastive unlearning paradigm, Clipout can unlearn target concepts in a more efficient and precise way.
Contributions. We propose the first machine unlearning method based on a contrastive objective with only negative samples considered. b) Utilizing contrastive learning theories, we provide necessary theoretical analyses on the effectiveness of the proposed unlearning method. c) Through extensive experiments, our proposed method not only exceeds previous sate-of-the-aae art methods in performance but also has significant superiority in efficiency.
2 Background
Contrastive objectives (Zbontar et al., 2021 Huang et al. 2023b) have been widely adopted in vision-
language pre-training tasks, and many VLMs use image-text pairs to calculate these losses. InfoNCE
(van den Oord et al., 2018) is one of the most canonical contrastive losses. Radford et al. (2021)
proposed CLIP to bridge vision and language domains by learning image and text encoders based on
InfoNCE, Following the idea of CLIP, a series of methods have been proposed (Jia et al.) 202 [ Li
al. 2021). J. For image generation tasks, Rombacc et al. (2022) proposed Stable Diffusion, which runs the diffusion process in the latent space to generate photo-realistic images. Stable Diffusion uses
fixed encoder to condition the images on the prompt.
Machine unlearning (Bourtoule et al. 2021 Ullah and Arora 2023 Tarun et al. 2023) allows trained models to discard specific information and maintain privacy by erasing the influence specific data points have on the models. Different from untearnable data (Huang et al., 2021 Liang et al. 2023) that safeguards personal privacy by adding the protective noise into data ahead of time, machine unlearning aims to modify a trained model to delete certain data points. While most methods focus on small-scale or discriminative tasks, some recent studies have started to pay attention to VLM scenarios, especially in unlearning Stable Diffusion (Heng and Soh 2023 Gandikota et al., 2024). Kumari al. (2023) proposed to ablate concepts in Stable Diffusion by overwriting them with general category anchor concept. Another concept-erasing method has been proposed by Gandikotta
et al. (2023). , which shares similar motivation behind compositional energy-based models (Du et al. 2020). Additionally, Schramowski et al. (2023) proposed an inference-time method, namely Safe Latent Diffusion, mitigate inappropriate concepts. Amara et al. (2025) discussed the ripple effects that may extend beyond the target concepts to erase. Kurmanji et al. (2023) proposed to use a contrastive-like loss to unlearn small nets for classification tasks, while we proposed to unlearn large-scale VLMs for image generation tasks. Generally, the text-to-image models intrinsically contain the text encoder for prompt conditioning. By unlearning the text encoder, our method can precisely remove the undesired concepts in the model, as the model cannot find the correct condition.
3 Contrastive Unlearning
We propose Clipout to unlearn private, copyrighted, or inappropriate concepts text-to-imaae VLMs. Based on the hypothesis of latent class (Wang and Isola, 2020 Huang et al. 2023a), the term concept in this paper refers to distribution that samples from the same concept share the same semantic meaning, and it is described by the prompt in text-to-image generation. We only unlearn
2

---

Unlearning the encoder smbedding
(3) Using the unleamned encoder
Negative Sumples
I-"childrea with guie"
Encader
2. 2."hiildren park"
"children with
Presininol Encrder
mh
Pre-rained
Generatoc
Figure I: The pipeline of Cl Clipout. Given prompt that describes concept to unlearn, Clipout first generates variants of the embedding by randomly masking the vector and then uses the contrastive objective to optimize the encoder. These masked vectors, e.g., Z; serve as negative samples in unlearning. The (red) positive samples, e.g., 2, are omitted in the optimization. The encoder is updated by maximizing the dissimilarity between these masked variants. After unlearning, the updated encoder replaces the original one, so the model can longer generate the removed concept.
part VLMs, namely the encoder that produces the embedding to generate conditions for other parts, and leave other parts fixed. Next, will introduce the objective used in Clipout with theoretical analyses. The following assumptions are made for simplicity. Proofs are present in the appendix.
Assumption 3.1. For any neural net fa(-) parameterized by 0 from a function class F, fo(-) is continuous. Likewise, we assume the similarity function sim(-) is also continuous and bounded.
Assumption 3.2. contrastive learning, the embedding and z from fo(-) share the same latent class if ||i z²11a is not too large, and with can form positive pair w.s. this latent class based on empirical studies (Goo al. 2021 By default, || denotes Frobenius or 22-norm.
3.1 Negative-Only Contrastive Loss
In text-to-image models, a concept is the semantic idea or category triggered by text prompt, which, in the context of representation learning, corresponds semantic distribution in the embedding space that ensures all conditioned generated samples share the same underlying semantics.
The gist of Clipout is make the concept embedding, which is used represent the conditions, unable to identify itself. Given a trained encoder so(-) and the target embedding So (xr) E Rd for an input data point x. randomly clip out some units from the embedding and set them zero Assume the mask m E {0, 13d, the contrastive loss, which inspired by InfoNCE, yields:
N
(0mm
to
log
N
esim (im(=SSm,smmm))
where is the batch size, T > 0 the temperature parameter, and sim(-) is the similarity function. Assume sim(2 O m. O m²) = have:
sim immm) 1
to
log
The element in is i.i.d. drawn from Bernoull(p)), and m can be viewed the vector of d Bernoulli trials. We term as the clipout rate analogous to the dropout rate (Srivastava et al. 2014). In single training epoch, la can be regarded the empirical version of the population loss:
to = m. log Σ
som
(3)
where m; and my are two independent random variables. According to Jensen's 's inequality, have:
Eg 2 1F my [sim(= @ m, ④ m,)] + log N 1.
(4)
lg 5 log Em,,, [e [eii((zmm.
log 1
As our goal is to minimize la. Eq. largely reduces to:
min Em [sim(= my, z m,)]
(5)

---

By generating multiple variants of 2, we treat these clipped embedding vectors O m as negative samples, and make them dissimilar from themselves. Since we do not use positive samples (i.e.. samples that should be gathered together) and only focus negative samples in the unlearning objective, is implied that the temperature learning methods (Chen et al.) 2020 He et al. (020). The pipeline of our method is presented in Figure | Given a target prompt. requires independent optimization for unlearning, and no extra corpus is needed for fine-tuning.
not as important as in conventional contrastive
3.2 On the Effectiveness of Unlearning
An empirical justification for randomly masking the embedding vectors is that these masked vectors serve as the positive samples for the input data point contrastive learning (Gao et al., 2021 Xu al. 2023). As we treat masked vectors as negative samples, we anticipate effective analyses with theoretical guarantees. In this part, we first analyze the convergence of our proposed negative-only contrastive loss in Eq. ①1) and then demonstrate why Cl Clipout is effective, from the perspectives of alignment and uniformity (Huang al. 2023a Wang and Isola 2020).
Convergence. From an intuitive perspective, Eq. (5) demonstrates that by optimizing Lo, the representation of v for a particular concept can be destroyed and it no longer represents its original semantic meaning. Besides, we can obtain the convergence guarantee of la as Proposition 3.3
Proposition 3.3 (Convergence). Given the temperature i..dd. and sim(= @ ma, @ m²) for all to in Eq. 1 converges to:
and the batch size N, if [m are
lim (Cg log N) Em, [og Emy (e (0mn(=00mm3mmm)// N-
Compared with theoretical results in previous work (Huang al. 2023a Wang and Isola 2020), Proposition implles that our loss converges to a point where the distribution of m matters, as there no positive sample in our loss. Since we only consider variants of single target embedding, the batch size N (i.e., the number of variants) can be (roughly) interpreted as the number of iterations in the optimization. Proposition 3.3 has ensured that the loss considered in Clipout tends to be convergent after masking the target embedding infinite times, and we now demonstrate the speed of convergence in Corollary 3.5
Lemma 3.4 (The upper bound of B E Wang and Isola 2020)). Given i.i.d. random
variable iy with bounded support C (- a, a). zero mean and o < a² variance, the expected value N
E [1 E xi]] is bounded by ((7)))
Corollary 3.5 (Convergence speed). The error term decays O ( w.r.t. the convergence speed of the limit in Proposition 3.3 according to Lemma 3.4
Corollary 3.5
encourages more negative samples in the optimization. As the dimension of vectors is usually compact, it is feasible to involve large batch size in practice.
Unlearning the target concept. To investigate whether Clipout can unlearn the target concept while limiting the bad impact on other concepts, we try to analyze the problem from the perspective of alignment and uniformity, as in Definition 3.6
Definition 3.6 (a-aligmment and B-uniformity (Huang et al. 2023a Wang and Isola 2020)). In the contrastive objective, alignment denotes positive pairs should be mapped to nearby features, and uniformity represents feature vectors should be uniformly distributed on the unit hypersphere to preserve the data information. Assume that features are normalized, i.e., [1](-)]12 = these are defined as follows:
Catign (f: En,x ((x))))
C (f: = log E Er. [e- (ells
(8)
where a >> 0,3 >>>0000000 O, a with x are positive samples, and x with x' are i.i.d. drawn from the input data distribution.

---

Algorithm Unlearning the encoder via Clipout
Parameter: prompt and encoder (-)
I: for it 1, altraaion do
2: Compute the data embedding fo(x);
masked samples from 4: Compute the contrastive loss Es w.r.f. Eq. (I)
Sample minibatch
5: Update 0 by descending the gradients: Vela;
6: end for
7: return unlearned encoder parameters 0":
Remark 3.7. Given a pre-trained encoder, alignment and uniformity be used to measure the unlearning performance. To unlearn a target concept. the unlearned encoder supposed to have poor alignment w.E.t the target, as poor alignment means deteriorated representation of the target. Meanwhile, the unlearned encoder should have good uniformity for any pair sampled from the input distribution, because the maximal information in the feature space should be preserved to maintain the model performance on unrelated tasks w.zl. other concepts.
Consider the cosine similarity in Eq. (). we have:
Em,,m sim O m,, 2 C my)]
② m4
2 2 my
1 E Em..my [I|
me]l
To minimize the loss, Eq. (9) implies that it needs to maximize the angle between two randomly masked embedding vectors, regardless the magnitudes. Previous kinds of literature (Gao et al.)
2021
Xu et al. 2023) have empirically proved that masked variants can be viewed positive samples.
Consider a = 2 in (7). we build the connection between the unlcarning objective and the loss of
alignment in Definition 3.6
(So; Emy,my [sim(2 @ m,, = @ m3)] = 2 (So; 2).
(10)
As in Definition 3.6 to learn a good representation, conventional contrastive learning aims at minimizing alignment positive samples, i.e., gathering these samples with the same latent semantics closely in the feature space. On the contrary, Eq. (IO) suggests our proposed unlearning objective intends to maximize the target's alignment loss by increasing the Euclidean distance between their unit vectors on the feature hypersphere. As a result, a pair of masked variants of z tends to become two vectors in opposite directions. By making learned representation scattered on the feature space, Cl Clipout can effectively unlearn the target concept by undermining the latent representations.
Influence different concepts. Naively, one can always unlearn the target by adding noise the model parameters. However, arbitrarily changing the parameters will make the model useless for other concepts. When maximizing the target alignment, the uniformity loss should not increase.
Lemma 3.8 (Uniform). For any x i.i.d. drawn from the input distribution, if the distribution of fo fa()) the uniform distribution aa 0 forms the minimizer for Euniform Joo 3).
Remark 3.9. For the unlearned encoder, we hope that the unlearned encoder with changed parameters still preserves much information as possible. This requires the loss converge point where the unlearned encoder can roughly form uniform distribution on the feature hypersphere.
As we have demonstrated that optimizing Eq. TT) helps to unlearn the target embedding, according to Lemma 3.8 we can alleviate side effects other concepts if the normalized feature distribution the uniform distribution. Consider normalized vectors u and v where u, v E R2d ||w|| = 1 and
||l|| = 1. Taking uTv m
23m,
=0my
for Eq. (6) in Proposition 3.3
we have:
[som, I
Em,
llo Em, [ [edintsom..omm///T
(11)
Em, logg Em, [e [/"]
Eq. ( can be largely reduced to Eq. (1) Lemma 3. 1. with proper clipout rates, since we mask these vectors first and then normalize them. Note that Eq. (1I) can only ensure uniform distribution over the masked embedding vectors. To maintain the maximal information at the global level, we have to ensure the normalized feature vector 2/||²||
||| is uniformly distributed on the unit hypersphere.
5

---

Table 1: Numerical results on VGGFace2 Cao et al. 2018). ) The sks person introduced by different personalized methods (e.g., DreamBooth (Ruiz et al. (023) is supposed to be unlearned, and similar concepts, such as female persons, ought to be kept. and indicate that the higher and lower values denote better performance, respectively. The prompts a photo of sks person, a photo female person, and photo of male person are used. Diff. represents the difference between the metrics of the target concept and the related concept. Values inside the bracket denote the difference compared with the results in the original pre-traie pre-trained model.
CLIP Score
FDFR
Textaal Inversion
(4)
male
DifL. (?)
sks (1)
male (1)
Diff. (1)
sks (1)
male
Diff. (U)
FSMG
26.04 (+0.18)
25.81 (4.08)
0.23 0.25 (30.23) 0.08 (+0.00) 0.17 (+0.11) 0.30 (.0.30) 0.10 (+0.00)
(-0.30)
ASD
23.63 (-2.23)
25.84 (-0.05)
2,21 18) 0.58 (+0.56) (-0.08) 0.42 (+0,36)
0.39 (-0.21)
0.22 (+0.12)
0.17 (-0.35)
23.21 (-2.65)
25.44 (40.55)
2.23
0.11 (+0.09) 0.12 (+0.0) 0.01 (-0.05)
(-0.4)
(40.01)
(40.3)
SLD
22.60 (-3.26)
25.45 (-0.44)
2.85 (+2.82) 0.52 (+0.50) 0.27 (+0.19)
0.25 (+0.19)
-0.01 (-0.61)
-0.03 (-0.13)
0.02 (40.48)
Clipout (Ours)
19.94 (-5.92)
26.06 (+0.17)
6.12 (+6.09)
0.80 0.11 (+0.03)
0.69 (+0.63)
0.13 (-0.47)
0.13 (+0.03) 0.00 (-0.50)
DreamBook
())
fomale (t)
DifL. (t)
(1) female (1)
Dift. (1)
sks (1)
fomale
Diff. ())
FSMG
27.25 (-1.7))))88)
(+0.20)
(31 0,76 (+0.70) 0.03 (+0.05) 0.73 (+0.67) 0.29 (-0.37) 0.22 (-0.06) 0.07 (-0.31)
ASD
28.21 (40.89)
27,80 (40.75)
0.41 (+0.14) 0.03 (-0:03) 0.03 (+0.05) 0.00 (-0.06)
0.55 (-0.11)
0.27 (-0.01)
0.28 (-0.10)
25.20 (:2.12)
26.71 (-0.34)
1.51 (+1.24) 0.33 (+0.27) 0.01 (+9.01) 0.32 (+0.26)
(-0.12)
(-0.10)
(-0.32)
SLD
25.65 (-1.67) 26,83 (.0.22)
1.18 (+0.91) (+0.88) 0.02 (+0.02)
0.42 (+0.36)
0.31 (.0.35)
0.04 (.0.24)
0.27 (40.1))
pout (Ours)
23.23 (-4.99) 27.15 (40.10)
3.92 (43,65 (.9.()))88) 0.03 (+0.03) 0.91 (+0.85)
(-0.33)
0:31 (+0.03)
(+0.36
LoRA
(4)
made
Difl. (f)
sls (t)
made (U)
Dift. (t)
ska (1)
male
Diff. (U)
ISMG
26.49 (+1.56) 26.40 (+0.48) 0.09 (:090) 0.38 (+0.36 0.02 (+0.00 0.36 (+0.36) 0.29 (40.34) 0.10 (-0.5) (40.9)
ASD
26.02 (+1.09) 26.26 (+0.34)
0.24 (-0.75) 0,56 (+0.59) 0.02 (+0.00) 0.54 (+0.54) 0,27 (-0.36)
0.21 (+0.06)
0.06 (-0.12)
24.13 (-0.80)
25.95 (+0.03)
(+0.83) 0.13 (+0.11) 0.02 (+0.00) 0.11 (+0.11)
(-0.23)
(+0.01) 0.24 (-0.24)
SLD
25.55 (+0.62) 26.27 (+0.35)
0.72 (.0.2) 0,06 0.19 (+0.17) (:0.1)) 0.31 (.0.32)
0.02 (-0.13)
0.29 (40.19
Cpou (Ours)
(-2.37) 27.01 (+1.09) 4.45 (43.46) 0.95 (40.93) (+0. 0.8 (+0.81) 0.10 (-0.53)
0.08 (-0.07) 0.02 (-0.4)
Lemma 3.10 (Minimizer Wang and Isola 2020 Given the hypersphere S = {= E Rd:
|| = 1}, and u, u. E 8d-1, μ E MiS 1) where M(+) is the set of Borel probability measures,
consider the following formulation:
e II dd(())dp(u))
min MEM(SS-1)
log
(12)
S
gd-1
where the uniform distribution a is the unique minimizer.
In Proposition 3.11 we show that if the masked (normalized) vectors in Eq. 11 are uniformly distributed on the unit hypersphere, the original normalized vectors are also uniformly distributed, implying that the unlearned encoder can preserve maximal information with our loss, making other concepts uniformly distributed and largely unaffected.
Proposition 3.11 (Feature distribution). Let IR& be a feature vector with independent components sampled from a distribution G. Let m E {0, 1]ad be a binary mask vector. Let S = (i m, = 1 be the set of active indices. If. conditioned on any S with (S] 2 2, the normalized sub-vector 2s/ || uniformly distributed on the unit sphere S then:
The feature distribution G is a spherical Gaussian distribution, i.e., 2 ~ NN 021 for some 0.
* The normalized,eattue vector 2/||æ|| is uniformly distributed on S
The implementation of Clipout is present in Algorithm 1 Our method does not rely on any label or data distribution of the original training data, and the negative samples are generated via a bootstrapping-lik like manner, i.e., masking the embedding vector itself.
Compared with previous methods, we do not need the anchor concept to align with (Kumari et al. 2023), 9, nor do we need the knowledge of downstream generators to select the specific layers for modification (Gandikota et al. 2023), and we indeed change the model parameters (Schramowski et al. 2023), which is hard to bypass. We only require minimal prior knowledge, which makes method more practical.
Besides, the efficiency of our method essentially results from the decoupling of complex text-to- image systems, and we only focus on the part that accounts for generating the condition. These demonstrations are verified in empirical evaluations.

---

Textual Inversion
DreamBooth
LoRA
Original
Unlearned
Original
Unlearned
Original
Unlearned
photo
portrait
photo
portrait
photo
portrait
photo
portrait
photo
portrait
photo
portrail
Figure 2: Face variations on CelebA-HO -HQ (Karras al. 2018) (the first two rows) and VGGFace2 (Cao et al. (2188) (the last two rows). Figures in the first and third rows are the target concepts to be unlearned, and figures the second and fourth rows are the similar concepts that we expect to maintain after unlearning. photo denotes the prompt "a photo of sks person" and portrail represents "a DSLR portrait of sks person". For similar concepts, the word sks is replaced by male or female.
4 Evaluation
We assess Clipout variety of tasks and datasets: the face datasets CelebA-HQ (Karras et al., 2018) and VGGFace2 (Cao et al. 2018). where the adversary could use personalized methods, e.g.
Textual Inversion (Gal et al. 2023), to make models remember personal concepts and forge fake
photos; b) LAION-5B (Schuhmann et al. 2022), where Stable Diffusion (Rombach et al. 2022) is pre-trained on and we make use of this dataset evaluate built-in concept unlearning. We consider Stable Diffusion in experiments, as it is significant open-source text-to-image model, and has enlightened many works (Zhang and Agrawala, 2023).
2021 is used to check the generated images match the prompt that describes them. We use Face Detection Failure Rate (FDFR) (Deng al. (2020) and Identity Score Matching (ISM) (Deng et al.)
Metrics. For text-to-image tasks, the metric of CLIP Score (Hessel al.
2022) measure the generated face quality compared with the training data. Once there is a face detected (i.e., the value of FDFR is not 1), the similarity between the generated images and the original ones the training dataset is computed for ISM. ArcFace recognizer (Deng et al. 2022) is used to compute the metrics of FDFR and ISM, and the pre-trained weights downloaded from its official code base.
Baselines. Since focus the prevention of unexpected results (e.g., model-related copyright issues) from well-trained models, for text-to-image generation, C1 Clipout is compared to three state- of-the-art concept removal methods: Ablating Stable Diffusion (ASD) (Kumari et al. 2023). ) Erasing Stable Diffusion (ESD) (Gandikota et al.) 2023), and Safe Latent Diffusion (SLD) (Schramowski al. 2023). These methods aim prevent the misuse of Stable Diffusion, and we report the best empirical results in each experiment, based on their public official implementations. Additionally, we also compare Clipout with FSMG (Le et al. 2023). which is advanced deepfake defense method based on unlearnable data by using the protective noise. Note that both SLD and FSMG operate in scenario different from the machine unlearning setting (Bourtoule et 2021).
Personal concept. Personalized techniques allow text-to-image models to learn new concepts given a few reference images. However, after crawling personal pictures from social platforms, the adversary could generate counterfeit photos via these techniques spread mendacious news.
In response to this privacy and safety concern, we investigate Clipout under three widely- personalized methods Stable Diffusion: Textual Inversion (Gal et al. 2023), which transfers
7

---

children with guns (Laget)
children at park
children with balloons
g
policemen with guns
maked woman (target)
var
well-dressed woman
&hrontiful woman
Disturbing pre-trained concept unlearning by Clipout. As modern VLMs are trained on scaled datasets, some built-in concepts in trained models may be offensive. Images are randomly sampled from the generator. The prompts "children with guns" and "a naked woman" describe the target concepts
Figure 3:
want to unlearn, and other harmless concepts are supposed to be kept. Images in the first and the second rows denote the results by the original and unlearned models, respectively. For naked experiments, use "a photo of" the prefix to generate realistic images. The stripes are manually added for publication.
new knowledge into models by learning a new word for the concept; b) DreamBooth (Ruiz et al. 2023), which fine-tunes models to learn concepts, and c) LoRA (Hu et al. 2022). which utilizes learned low-rank matrices to represent new concepts. The results are depicted in Figure 2 Based on generate photo-realistic facial images according to text prompts. We use Adam (Kingma and Ba 2015) as the optimizer with learning rate of 1.5 x 10-5 and perform unlearning for 200 epochs. The clipout rate is 0.25 by default. After being unlearned by our method, the transferred concepts of the particular person no longer exist, and the resultant images become meaningless (e.g. random scenes) while keeping the ability of models to generate figure-like images.
transferred concepts from VGGFace2 and CelebA-HQ Stable Diffusion
We compare Clipout with baselines and report numerical results in Table T For original and unlearned models, we evaluate metrics based on 128 generated images in each experiment. As CLIP Score and FDFR denote whether the picture conforms to the text and whether it fails to detect the face, we expect to enlarge the difference between images from the target and related concepts and reduce ISM to differentiate the generated images from the source. It is observed that Clipout significantly outperforms baselines about CLIP Score, FDFR, and ISM in all experiments. Considering the difference between images from the target concept and related concepts, our method's CLIP Score is more than double that of the baselines, and the results of FDFR imply that Cl Clipout only has minimal side effects on other related concepts. As for ISM, results indicate that after unlearning, the generated images do not look like the original reference images, which is consistent with the visual results in Figure The experiments indicate that our proposed method can precisely unlearn the target concept of particular person while keeping the similar concept as is.
Concept pre-trained models. Benefit from an enormous amount of training data from the Internet, text-to-image models enjoy a high quality of generated pictures. Yet disturbing concepts may be unintentionally implanted into pre-trained VLMs, due to the unfiltered training data.
In this part, we experiment with built-in concepts concerning Stable Diffusion, which is pre-traue treitra on the subsets of LAION-5B. We manage to unlearn two disturbing concepts, namely children with guns and naked woman, and inspect whether other related concepts are affected by the unlearning process. Results are presented Figure 3 The VLMM unlearned by CL alipout cannot produce images conditioned on disturbing concepts, e.g., the combination of naked and woman, anymore, and benign

---

Original
25% Promps Unlearmed
50% Prompt Unlearned
73% Prompt Unlearmed
Full Prompt Unleamed
Figure 4: On matching the long target prompts in style transfer. Images in the first row oriented from the prompt Claude Monet inspired painting / of female botanist / surrounded by exotic plants in greenhouse, which is the full target prompt to unlearn. We use the partial prompt in the unlearning optimization and gradually lengthen the prompt by / to match the full target prompt.
377
502
37311
0.27 027
0.55
0.0
6.00
300
S77
Usat
2000C
664
4.0
0.53
6113
10413
10433
200
207
1.00
270
0.27
0667
6.50
(a) Time Efficiency
(b) GPU Mem. Efficiency
(c) CLIP Score
(d) FDFR
Figure 5: (a)(b) Efficiency evaluation on the time usage and GPU memory usage. NVIDIA A40 is used to conduct evaluations. Baselines are based on their official implementations. (c) Ablation studies on the learning rate (vertical) and clipout rate (horizontal) for unlearning. We benchmark Clipout on VGGFace2 (Cao et al. (018). The absolute value of the difference between the metrics of the target concept and the related concept is reported. FDFR and CLIP scores are present as evaluation metries. The higher score denotes better unlearning performance, i.c., the target concept is unlearned while the related concept remains as is.
concepts like naked woman alone remain intact. There is no observed reduction in the diversity of other concepts or biases introduced by unlearning. Results show that Clipout can effectively unlearn the target and retain other concepts to the greatest extent.
Unlearning built-in styles. The prompt standing for the target concept or used in practice may not be exactly the prompt used in the unlearning process. We try to answer whether Clipout is effective without an exact match in the prompt. We consider the style transfer scenario, where this issue is likely happen, as detailed paintings need more words describe them. As in Figure 4 two prompts. "Claude Monet inspired painting of female botanist surrounded by exotic plants in greenhouse" (i.e., Claude Monet) and "Pablo Picasso inspired painting of a ship sailing in a stormy sea with dramatic lighting and powerful waves" (i.e., Pablo Picasso), are evaluated. We gradually lengthen the prompt to match the full prompt of Claude Monet and examine the generated paintings. Results show that the unauthorized style imitation can be prevented with the short prompt "Claude Monet inspired painting" unlearned, and other styles can be largely maintained.
Efficiency evaluation. As Clipout only unlearns a part of the large VLM, our method enjoys high efficiency. We compare Clipout with basclines in terms of efficiency by measuring the average usage in Table [ with the same empirical setup, and the results are plotted in Figure 5) It is noted that Clipout is far more time-efficient than ASD and ESD, and only has a minimal GPU memory requirement. We ascribe this efficiency to the trait that Clipout decouples the large system into different artifacts, and only focuses on the artifact obligated to generate the concept condition.
Additionally, inference-ting mitigation methods like SLD are training-fre free and also enjoy high efficiency. However, standard machine unlearning implies making model forget or erase knowleds ledge from its parameters. SLD does the opposite: it actively relies the model's already acquired knowledge of inappropriateness. It then uses this knowledge to apply the safety guidance during the diffusion process, which suppresses removes inappropriate image parts they are being generated.

---

Therefore, it is more precise to demonstrate training- free methods like SLD as training-free free safety guidance or content suppression technique that steers generation away from unsafe concepts rather than one that makes the model forget them.
Hyperparameter tuning. Due to the high efficiency of our method, we can conveniently tune the parameters and investigate the factors that contribute to the unlearning performance. The heat maps are shown Figure [ The degeneration of performance can be seen if C1 ipuut clips no unit or all units out of the embedding, which demonstrates the validity of randomly clipping some units out of the concept embedding. Besides, if we increase the learning rate during the unlearning process, all concepts suffer from deterioration. Contrarily, with small learning rates, the target concept and other concepts will not be significantly affected. This shows that it is vital to choose a reasonable learning rate for unlearning, while Cl ipout is insensitive to the ratio of clipped units within certain range.
Robustness analysis. We conducted adversarial prompt tests using Ring-A-Bell Tsai et al. 2024 model-agnostic red-teaming framework that automatically generates optimized adversarial prompts to probe safety mechanisms in text-to-image diffusion models. Results are reported as follows: For the model trained on VGGFace2 using DreamBooth, FDFR for the target concept 0.06, while for the benign female concept, the metric stays at zero. With Clipout, FDFR for the target concept rises to 0.94 while the benign concept remains low at 0.03. When adversarial prompts from Ring-A- are applied to Clipout, FDFR for the target concept remains high at 0.58, with the benign concept still at 0.03. These results confirm that Clipout maintains strong erasure of the targeted concept and preserves benign concepts even under adversarial prompting. albeit with some degradation relative to clean prompts. While Ring-A Laigg-A-Blll Bell is designed to produce forged, highly optimized prompts that are often unnatural in real-world usage, this evaluation complements our core threat model, in which malicious users most commonly attempt to recover erased concepts via natural language variation like synonyms, metaphors, and cultural paraphrases, rather than obfuscated tokens.
5 Broader Impact
Our work on machine unlcarning for text-to-image diffusion models has both positive and negative societal impacts. On the positive side, selective unlearning can reduce exposure to harmful content (e.g., nudity or violence), honor take-down and consent-withdrawer requests, mitigate leakage of sensitive or copyrighted material, and enable targeted safety updates without full retraining, contributing to safer digital spaces. On the negative side, the same capability can enable censorship, erase culturally significant or contested concepts, and introduce or amplify bias depending on who defines inappropriate (Hall et al. 2023: Berg et al.) 2022 Hall et al. 2022 Agarwal et al.) 2021 Hong et al., 2024)). may also be misused. For example, adversaries could use CI Clipout to remove watermarks inside the trained models (Zhao et al.) 2023b Liu et al.) 2023 Zhao et al., 2023a) and falsely claim ownership or copyright. Our aim is to foreground copyright and privacy issues in large VLMs and promote awareness and responsible practice within the machine learning community.
6 Conclusion
In this paper, we demonstrate the potential risk in the text-to-image VLMs and propose Cl Clipout based on the negative-only contrastive loss to alleviate these risks. By unlearning target concepts in text-to-image models, we purify the trained VLMs with negligible costs. We believe the proposed method will play an indispensable role in building a more responsible text-to-image system, as its efficacy has been demonstrated for removing undesired concepts.
There are still several limitations with our Clipout. Similar other deep learning approaches, the empirical results of our proposed method rely on the choice of the hyperparameter. Besides, since the negative samples are generated via a bootstrapping-liie manner, the unlearning direction of the proposed method is not controllable, which makes the generated images usually look like some landscapes or texture patterns. This unpredictability highlights the need for further refinement to achieve more controllable unlearning outcomes.
Acknowledgment: We would like to acknowledge the valuable insights and suggestions provided by the anonymous reviewers.
10

---

References
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating CLIP: towards characterization of broader capabilities and downstream implications. CoRR. abs/2108.02816 2021.
Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajie, Zarana Parekh, Natalic Harris, Sarah Young. Chirag Nagpal, Najoung Kim, Junfeng He, Cristina Nader Vasconcelos, Deepak Ramachandran, Golnoosh Farnadi, Katherine A. Heller, Mohammad Havaci, and Negar Rostamzadeh. Brasehench: Understanding the ripple effects of concept erasure techniques. CoRR, abs/2501 100833,, 2025.
Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Hannah Kirk, Aleksandar Shtedritski, and Max Bain. prompt anay keeps the bias away: Debiasing vision-language models with adversarial learning. In AACL/IJCNLP (1), pages 806-822, Association for Computational Linguistics, 2022.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. CoRR, ab./2110.01963, 2021.
Salomon Bochner. Monotone funktionen, sticltjessche integralc und harmonische analyse. Mathematische Annalen, 108(1):378-44) 1933.
I Lucas Bourtoule, Varun Chandrasckaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Paperot. Machine unlcarning. In SP, pages 141-159. IERE, 2021.
Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In FG. pages 67-74. IEEE Computer Society, 2018.
Ting Chen, Simon Kornblith. Mohammad Norouzi, Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
Zhikai Chen, Lingxi Xic, Shanmin Pang, Yong Hc, and Bo Zhang. Magdr: Mask-guided detection and reconstruction for defending deepfakes. In CVPR, pages 9014-9023, Computer Vision Foundation IEEE, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, pages 2818-2829. IEEE, 2023.
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, pages 5202-520 5211. Computer Vision Foundation IEEE, 2020.
Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. IEEE Trans. Pattern Anal. Mach. Intell., 5419599596 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x 16 words: Transformers for image recognition at scale. In JCLR. OpenReview.net, 2021.
Yilun Du Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In NewrIPS, 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-O Or. An image worth word: Personalizing text-to-image generation using textual inversion. In ICLR. OpenReview.nec, 2023.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto Kaufman, and David Bau. Erasing concepts from diffusion models. In ICCV, pages 2426-2436. IEBE, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing in diffusion models. In WACV, pages 5099-5108. IEEE, 2024.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP (7), pages 6894 6910. Association for Computational Linguistics, 2021.
Melissa Hall, Laurens van der Maaten, Laura Gustafson, and Aaron Adcock. A systematic study of bias amplification. CoRR, abs/2201.11706, 2022.

---

Siobhan Mackenzie Hall, Fernanda Gonçalves Abrantes, Hanwen Zhu, Grace Sodunke. Aleksandar Shtedritski, and Hannah Rose Kirk. Visogender: dataset for benchmarking gender bias image-text pronoun resolution. In NeurlPS, 2023.
Jeff 7. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurlPS, pages 5000-5011, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726-9735. Computer Vision Foundation IEEE, 2020.
Alvin Heng and Harold Soh, Selective amnesia: A continual learning approach to forgetting in deep generative models. In NeurlPS, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forhes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP (1), pages 7514-7528. Association for Computational Linguistics. 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NIPS, pages 6626-66666 07766-667722 2017.
Rachel Hong, William Agnew, Tadayoshi Kohno, and Jamie Morgenstern. Who's in and who's out? A case study of multimodal clip-filterig in datacomp. In EAAMO, pages 4:147:17. ACM, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022.
Hanxun Huang. Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In /CLR. OpenReview.net, 2021.
Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. In ICLR. OpenReview.net, 2023a.
Zizheng Huang, Haoxing Chen, Ziqi Wen, Chao Zhang, Huaxiong Li, Bo Wang, and Chunlin Chen. Modcl-aware contrastive learning: Towards escaping the dilemmas. In ICML, pages 13774-13799 PMLR, 2023b.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quac V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904 4916. PMLR, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR. OpenReview.net, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR (Poster), 2015.
Nupur Kumari, Bingliang Zhang, Sheng-Yo Yu Wang. Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating IEEE, 2023.
concepts in text-to-image diffusion models. In ICCV, pages
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlcarning. In NeurlPS, 2023.
Thanh Van Le. Hao Phung. Thuan Hoang Nguyen, Quan Dao, Ngoc Tran, and Anh Tran. Anti-dreammooth: Protecting users from personalized text-to-image synthesis. In ICCV, 2023.
Junnan Li, Ramprasaath R. Selvaraju. Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, pages 9694-9705, 2021.
Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. In NeurlPS, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified languael understanding and generation. In ICML, pages 12888- -2900. PMLR, 2022b.
Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. In ICML, pages 20763-20786. PMLR. 2023.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsofi COCO: common objects in context. In ECCV (5), pages 740-755, Springer. 2014.
12

---

Hanwen Liu, Zhenyu Weng, Yuesheng Zhu, and Yadong Mu. Trapdoor normalization with irreversible ownership verification. In ICML, pages 22177-22187. PMLR, 2023.
Yufan Liu, Jinyang An, Wanqian Zhang. Ming Li, Dayan Wu, Jingzi Gu, Zheng Lin, and Weiping Wang. Realera: Semantic- levell concept erasure via neighbor-coneep mining. CoRR, abs/2410.0914,, 2024.
James Clerk Maxwell. Ii. illustrations of the dynamical theory of gases. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 20( 20(130):2 (10):211-37 1860.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, pigh-perfrrmance deep learning library. In NeurIPS, pages 8024-8035, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and lly Sutskever. scro-shot text-to-image generation. In ICML, pages 8821-8831. PMLR, 2021.
Robin Rombach, Andreas Blatumann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685 IEEE, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch. Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pages 22500-22510. IEEE, 2023.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICMI, pages 5628-5637. PMIR, 2019.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffuusion: Mitigating inappropriate degeneration in diffusion models. In CVPR, pages 22522-22531. IEEE, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo
Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Rohert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NewrIPS, 2022.
Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In CVPR, pages 18699-18708. IEEE, 2022.
Nitish Srivastava, Geoffrey F. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., (((1)):199-1999))) 2014.
Matteo Stefamini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From show to tell: A survey on deep learnin-bbased image captioning. IEEE Trans. Pattem Anal. Mach. Intell., 45(1)5539-559, 2023.
James Stewart. Positive definite functions and generalizations, an historical survey. The Rocky Mountain Journal of Mathematics, 6(3):409 434, 1976.
Ayush Kumar Tarun, Vikram Singh Chundawat, Murari Mandal, and Mohan S. Kankanhalli. Deep regression unlearning. In ICML, pages 33921-33939. PMLR, 2023.
Jiaye Teng, Weiran Huang, and Haowei He. Can pretext-based self-supervised learning be boosted by downstream data? A theoretical analysis. In AISTATS, pages 4198-4216 PMLR, 2022.
Yu-Lin Tsai, Chia-Y Yi Hsu, Chulin Xie, Chih- Hsun Lin, You Chen, Bo Li. Pin- Yu Chen, Chia-Mu Yu, and Chun- Chun-Ying Ving Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? In ICLR. OpenReview.net, 2024.
Enayat Ullah and Raman Arora. From adaptive query release to machine unlearning, In ICML, pages 34642- 34667. PMLR, 2023.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 007.1807.03748. 2018
13

---

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In ICML, pages 9929-9939. PMLR, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer, Patrick von Platen. Clara Ma. Yacine Jernite, Julien Plu. Canwen Xu, Teven Scao. Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ue natural language processing. In EMNLP (Demos), pages 38-45. Association for Computational Linguistics, 2020.
Yongliang Wu, Shiji Zhou, Mingzhuo Yang. Lianzhe Wang, Wenbo Zhu, Heng Chang, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. CoRR, abs/2405. 15304, 2024.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlci Zhou, and Philip S. Yu. Machine unlcarning: A survey, ACM Comput. Surv. 56(1):9:1-9:36 2024.
Jiahao Xu. Wei Shao, Lihui Chen, and Lemao Liu. Simese++: Improving contrastive learning for sentence em- beddings from two perspectives. In EMNLP, pages 2302--12040 Association for Computational I. Linguistics, 2023.
Peng Xu, Wenqi Shao, Kaipeng Zhang. Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng. Siyuan Huang. Yu Qiao, and Ping Luo. Lvlm-chub: comprehensive evaluation benchmark for large sainn-la models. JEEE Trans. Pattern Anal. Mach. Intell., 47(3):1877-1893, 2025.
Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. ARCANE: an efficient architecture for exact machine unlearning. In IJCAI, pages 4006 4013. ijeai.org. 2022.
Michihiro Yasunaga, Armen Aghajanyan. Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tai Yih. Retrieval-augmented multimodal language modeling. ICMI., pages 39755-39769. PMLR, 2023.
Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. Artificial fingerprinting for generative models: Rooting deepfake attrihution in training data. In ICCV, pages 14428-14437. IEEE, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Sclf-supervised learning via redundancy reduction. In ICML, pages 12310- PMLR, 2021.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. CoRR. abs/2302.05543, 2023.
Xuandong Zhao, Yu- -i-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In ICML, pages 42187-421 42199, PMLR, 2023a.
Yunqing Zhao, Tianyu Pang. Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusion models. CoRR, 0ttp//333..0333 2023b.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Please see Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Appendix
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification: The data and codes are not released at submission time. For experimental result reproducibility, setting and details, please see Appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It would be too computationally expensive for baseline methods, and all the numerical results are based on 128 randomly generated images for statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focuses machine unlearning, which aims to remove particular concepts in the trained models, instead of releasing large model with a high risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Existing assets properly credited and cited in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets at submission time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A Proofs
Following previous work in contrastive learning theories (Saunshi al. 2019; Huang et al. 2023a
HaoChen et al. 2021 Wang and Isola 2020 Teng et al.) 2022), we present the full proofs as below.
Proposition A.1 (Convergence). Given the temperature > and the batch size N, if {m, J are i..dd and sim(z o m, ④ m,) for all ly in Eq. ① converges
lim (Cg log N) = Em, [og Em, [e time (0mm N -000
Proof. Using sim(= o m, O m;) 1 and Eq. 2 we have:
lim (Le - log N)
N
lim N000
log
N
T
Since sim(-) is bounded (Assumption 3.1 the term log N E sim(-)/T uniformly bounded. By the Law of Large Numbers, for each fixed m,:
slmm 0.8. ( [
N
Likewise, by continuity of log and the Bounded Convergence Theorem, have:
lim (Ly - log N) = Em, log Emy [cmm(=Gm..s0m/)/
N
Lemma A.2. Given y > and 2 > 0, log - log 2|| S E"Elly - 21 holds if e- = is a lower bound of both y and
Proof. Since 2 e- and = 2 6-=,,__ any & between and z satisfies g 2 e- and hence s see By the Mean Value Theorem, there exists such & with
1 y 2| Seeell 2/-.
log y log 2|
Taking expectations on both sides yields the result.
Corollary A.3 (Convergence speed). The error term decays in O ((+)) W.r.t. the convergence speed of the limit in Proposition 3.3 according to Lemma 3.4
Proof. According to Proposition 3.3 and Eq. the error term is:
N
Err(Eg log N) = Emm log Em, [e Smm log
mim(:Gm...0mmm
N
j=l
N
Eyn
[ |
log Em,
log
where the inequality follows from Jensen's inequality since | is convex.
Since sim(-) :) -1, both Em, [esi and N Σ/= clim(-)/+ are lower bounded by e 1/7, By
Lemma A.2
N Fmmyy (im _0t
Enr(eo log N) S cl//Em,
1=1
22

---

For fixed m,, the summands are i..d. with zero mean (by definition of expectation) and bounded
support since esim(-\/r E [e-1/r el/rJ. By Lemma 3.4
Err(Lo log N) S O VN
Lemma A.4 (Uniform). For any x: ii.d. drawn from the input distribution, if the distribution of fo fa( the uniform distribution aa forms the minimizer for Exniform B).
Proof. Since f-)-) ) is normalized, the loss Luniform (foi 3) can be expressed via the Gaussian kernel
Gg(u, =||²=+|1" By classical results (Bochner 1933 Stewart 1976). Gg g strictly positive
definite, and consequently aa uniquely minimizes e eusiiinn (fats) [Wang and Isola 2020). Since exp(
is monotonically increasing, dd also uniquely minimizes umiioomm P).
Theorem A.5 (Maxwell's Characterization of the Gaussian Distribution (Maxwell 1860 ). Let
X (X1, X2) be random vector in IR2 satisfying the following conditions:
1. Independence: X1 and X2 are independent random variables.
2. Spherical symmetry: The distribution of X is rotationally invariant, for any orthogonal mairix E 02), the random vectors QX and X have the same distribution.
Then X1 and X2 are either both identically zero, or both normally distributed with zero mean and identical variance. That is, either X almost surely, or X ~~NN "0, 12)
1 for some a >
Proposition A.6 (Feature distribution). Let z E IRd be a feature vector with independent components sampled from distribution G. Let m {0, d be binary mask vector. Let S {i m, = 1} be the set of active indices. If. conditioned on any with [) 2 2, the normalized sub-vector zs/ [| uniformly distributed on the unit sphere S|S1-1, then:
The feature distribution G spherical Gaussian distribution, i.e., N/(0, for some a > O.
The normalized feature vector 2/||æ|| uniformly distributed on 1
Proof. Let S {i,s} be any pair of distinct indices. By the premise, the normalized vector (2,, 2,)/[[(z,, 2 is uniformly distributed on S Since the components of are i.i.d. (independent and identically distributed from G), it follows that the joint distribution of (zi; 2j) is rotationally invariant: the product density (())(()) must be constant on circles centered at the origin, hence depends only x2 y
By Maxwell's theorem (Thcorem A.5). independence together with rotational invariance implies that and zj normally distributed with zero mean and identical variance o² > 0. Since this holds for all pairs, all components of z are i.i.d. Gaussian, and thus 2 ~ (O0,, l2).
Finally, the spherical Gaussian distribution is invariant under orthogonal transformations, so =/ |||||| is uniformly distributed on Sd-1
Table 2:
The norm of the embedding. We calculate the norm of embedding vectors to investigate the dif- ference before and after unlearning among the target concept and similar concepts. Diff. denotes the differ- ence after unlearning the encoder.
Table Numerical results on the I2P
dataset (Schramowski et al., 2023). We focus on the "Sexual" category, including armpits, belly, buttocks, feet, breasts and genitalia.
Prompt Text
Original Unlearned
Diff.
Method
Unsafe proportion
children with guns (target)
-0.1699
-0.1677
0.0022
Stable Diffusion (Clean)
28.54%
children at park
-0.1696
-0.1680
-0.0008
SLD
14.97%
guns
-0.1697
-0.1692
0.0005
Clipout (Ours)
11.06 %
23

---

B Experiment Details and Discussions
In this section, we report implementation details, additional empirical results and discussions w.r.f. our proposed method.
Experimental setup. We use Adam Kingma and Ba 2013) as the optimizer with a learning rate of 1.5 x 10-5 and perform unlearning for 200 epochs. The clipout rate is set as 0.25 by default. For diffusion models, we use Stable Diffusion 2.1 (Rombach et al. 2022). with CLIP (Radford et al. 2021 Cherti al., (023) the text encoder. All pre-trained weights downloaded from the Hugging Face platform (Wolf et al. 2020). For numerical results, unless stated otherwise, we calculate these results w.r.t. different metrics based 128 randomly generated images with 512 x 512 resolution for statistical significance. The hyper-parameters in baseline methods are set as the recommended values described in their publications. For reproducibility, we set the same random seed for Numpy and PyTorcc (Paszke el al., 2019). We use PyTorch 1.13.1 with CUDA 11.6 on the Ubuntu operating system. NVIDIA A40 with 48 GB GDDR6 memory is used conduct most experiments. As is described in the main text, our proposed method is efficient, which makes it easy to tune the hyperparameters.
For datasets, we choose the face datasets CelebA-HQ (Karras et al., 2018) and VGGFace2 (Cao
et al. 2018). These face datasets commonly used in deepfake or privacy related tasks (L [Liang
et al. 2023 Le et al. (2023). We follow the same dataset split in the previous work (Le et al. 2023).
LAION-5B (Schuhmann et al., 2022) is also considered, where Stable Diffusion (Rombach al.
2022) is pre-trained on and we make use of this dataset to remove target concepts from pre-trained
large diffusion models.
On the norm of the target embedding. In unlearning experiments, we calculated the distribution of the embedding vectors, and it is observed that there is significant change w.r.f. the target embedding. Once the text encoder is unlearned, the embedding of a particular concept is fixed. We use 27 prompt templates such as photo of as in (Radford et al. [021) generate multiple embedding vectors centering around a concept. As in Table 2 for the target concept children with guns, the mean 1699 and 1677 before and after unlearning respectively,
values of embedding vectors and the average mean value is increased by 0.0022. The difference in the target concept embedding vector is much higher than that of related concept embedding vectors, which supports alignment and uniformity claims in the main text.
Personal and non- non-persone personal concepts. In the main text, we perform experiments personal concepts (e.g., sks person introduced by Textual Inversion) and non-personal concepts (e.g., children with guns and Claude Monet inspired painting), and it turns out that proposed method can unlearn both concepts. Besides, for Textual Inversion experiments, the personalized concept is merely a new embedding in the text encoder. Once the new and personalized concept is learned, there is no difference between the personalized concept and the non non-personalized concept, since the parameters w.r.f. other parts of the model are unchanged.
Inappropriate image prompt benchmark. We performed experiments on the inappropriate image prompts (I2P) dataset (Schramowski et al. 2023). and compared our proposed method with baselines. For Clipout, we performed evaluations on the "Sexual" category of 12P, with the classifier metric of NudeNet (Schramowski et al. 2023). For the baseline Safe Latent Diffusion (SLD) (Schramowski jet al. 2023), we keep its default settings. The empirical results are presented in Table 3 is noted that the default SLD setting includes many "Bexual"-relatte" keywords, including but not limited to Nudity, while our proposed method only unlearns a naked woman in the Stable Diffusion model, which can be considered as stress test for our Cl Clipout. The empirical results demonstrate that proposed method still outperforms the baseline method on the 12P dataset.
Other metrics for numerical results. For other metrics, instead of CLIP Score, FDFR and ISM in the main text, we also evaluate the FID Score (Heusel et al., 2017) on the target concept and related concepts. Empirical results are reported in Table For each experiment, we compute the FID Score between 512 images generated by original models and unlearned models, respectively. is observed that after unlearning, the FID Score for children with guns is 463. 14, which is much higher than that
24

---

Table 4: On the metric of FID Score. The score calculated between the images generated by the original en- coder and the images generated by the unlearned encoder.
Table 5: Empirical results synonymous words. The metric of ISM is reported. Chris Hemsworth who plays Thor in the movies is the target concept to unlearn. Note there are other characters playing Thor in other movies, comics or cartoons.
Prompt Text
FID Score
Prompt Text
Original Unlearned Diff.
children with guns (target)
463.14
Chris Hemsworth
0.95
0.22
-0.73
children with balloons
121.21
male person
0.33
0.23
-0.10
guns
133.12
the actor who plays Thor
0.75
0.50
-0.25
policemen with guns
135.22
the actor who plays Captain America
0.49
0.42
-0.07
of related concepts. Experiments demonstrate that our proposed method can effectively unlearn the target concept while preserving the related concept as is.
Inclusion relation between concepts. Note that there is the inclusion relation between certain concepts (e.g., children with guns and guns). After unlearning the concept of guns, the results show that some generated images of children with guns are reduced to that of children, yet some other generated images become meaningless images since the concept of guns is unlearned. In fact, this unlearning situation is unusual in practice, since the concept guns alone is not regarded as a disturbing concept in most cases, only when combined with other concepts like children are considered to be disturbing.
Unlearning multiple concepts. Multiple concepts can be sequentially unlearned, since unlearning one concept is an independent optimization process and other concepts can be largely maintained, regardless of how many concepts are unlearned in total. As in Figure 3, since unlearning the first concept (e.g., a naked woman) will not affect the other concept (e.g., naked mole rat), we can continue to unlearn other concepts in a sequential order. Therefore, the original generation capacity of the text-to-image model will not be largely influenced by the proposed method.
Synonymous words. When describing a particular concept, there may be multiple choices for the descriptive prompt text. We take "Chris Hemsworth" ("the actor who plays Thor") as the target to unlearn, and also evaluate two related prompts, namely "male person" " and "the actor who plays Captain America", to investigate whether our proposed method can precisely unlearn the target concept in pre-trained diffusion models. These non oon-personllize concepts are intrinsically involved in the pre-trained Stable Diffusion models. We try to unlearn if these celebrities can be removed from pre- pre-trained diffusion models. We first generate 128 images from the pre-trained Stable Diffusion as the reference images and calculate the metric of Identity Score Matching (ISM) measure the similarity between the generated images and the reference images. Numerical results are reported in Table It is observed that after unlearning, the similarity between generated Chris Hemsworth or Thor images and the reference images significantly decreases, while related concepts are largely unaffected. These results indicate that the proposed Cl Clipout works in the scenario of synonymous words.
Details on I2P benchmark. We performed experiments the inappropriate image prompts (I2P) dataset (Schramowski et al. 2023), and empirical results are reported in the main text. The I2P benchmark includes real user prompts for text-to-image generation that are likely produce inappropriate images. It is designed to assess mitigation strategies in Stable Diffusion, without being tied any specific model. Inappropriate content defined offensive, threatening, or anxiety- inducing, including categories like hate, harassment, violence, images, and illegal activity. These concepts vary across cultures and evolve over time. The benchmark uses 26 keywords and phrases to detail these categories, collecting up to 250 real-world prompts for each. These prompts generate images that align closely with inappropriate concepts in CLIP space.
sexual content, shocking
The category of sexual content is considered for experiments, which includes armpits, belly, buttocks,
feet, female breasts, male breasts, male genitalia and female genitalia (Liu et al. 2024 Wu et
2024).
25

---

Table 6: Results for diverse prompts. 50 different prompts are sampled from MS-COCO (Lin et al. 2014 for evaluation.
Table 7: Results for additional baselines. Diff. represents the difference between the metrics of the target concept and the related concept.
Method
FID Score
Method
FDFR (Diff.)
ISM (Diff..
Original
151.03
0.18 (+0.12)
0.11 (-0.27)
w/ "children with guns" Unlearned
157.79 (+4.48%%
UCE
0.25 (+0.19)
0.10 (-0.28)
w/ "a naked woman" Unlearned 163.52 (+8.27%)
Clipout (Ours)
0.91 (+0.85)
0.02 (-0.36)
Diverse prompts. We benchmark 50 diverse MS-COCO prompts (Lin et al. 2014) (e.g., "a dog catching a frishee") to check whether C1 Clipout degrades the model's overall generative quality on broader prompt set. Given prompt, the score is calculated between the images generated by the original encoder and the images generated by the unlearned encoder. Note that the metric of CLIP Score may be biased (Agarwal et al. 2021) (e.g., in terms of race, gender, and age). In this part, use FID Score as the metric, considering the diversity of sampled prompts. All images are randomly generated. The average score reported in Table Since the unlearning process aims change the model parameters, and the prompts may already contain or overlap with words in the target prompts, it is inevitable to see minor performance drops. Compared the FID score change with respect to the target prompt (e.g., Table results confirm that the overall generative ability of the model on unrelated prompts is largely preserved.
Additional baselines. We include empirical evaluations about additional baselines like SA (Heng and Soh 22023) and UCE (Gandikota al. 2024). The results are reported in Table In the table, values inside the bracket denote the difference compared with the results in the original pre-trained model. Numerical results indicate that Cl ipout remains competitive, particularly in effectiveness, specificity, and preservation of non-target concepts.
26


Reviews:
[{"review_id": "9lgHW5Wk9D", "reviewer": "Reviewer_ReZd", "summary": "The paper introduces Clipout, a unlearning method that removes specific concepts from pre-trained T2I models without retraining. This method randomly clips the embedding of the target concept and uses a contrastive objective to make these altered embeddings dissimilar from each other. The paper validates this method on personal concept removal and built-in concept removal.", "questions": "1.  The paper proposes to use a random clip of the embedding of the target concept. How about other methods to disrupt representation, like perturbing the target embedding with different levels of Gaussian distribution?\n2. Current research shows that the unlearning cannot resist the adversarial attacks. Could the authors analyze the robustness of the proposed method?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel solution for T2I unlearning with only negative samples. \n2. The paper validates across both personal concept removal and built-in concept removal, demonstrating its effectiveness. \n3. The paper shows that its method outperforms other baseline models with high computing efficiency.\n\nDrawback:\nThe major concern of the paper is that while the paper effectively demonstrates that Clipout preserves the model's ability to generate semantically related concepts, it lacks the evaluation of the model's performance on a broad set of common, unrelated prompts. The current experiments show that the unlearning is precise, but they do not sufficiently prove that it does not degrade the model's core generative quality overall.", "was_revised": true, "final_justification": "My concerns towards the paper are two folds: 1) whether the model maintains its general performance on unrelated tasks after the unlearning process, and 2) whether the proposed Clipout method is robust against adversarial attacks.\n\nWhile during the rebuttal, the authors provide some empirical results for each concern, I recommend including more comprehensive experimental results.", "rebuttal": "Thank you for your detailed review and thoughtful suggestions. We are grateful for your positive feedback on the novelty, soundness, and contributions of our paper. Below, we respond to your comments to clarify misunderstandings and alleviate your concerns.\n\n**Q1:** The paper focuses on semantically related concepts but does not evaluate the model's performance on a broad, unrelated prompt set. Thus, it is unclear whether Clipout degrades the model's overall generative quality.\n\n**A1:** This is an excellent point, and we agree that demonstrating minimal degradation on general generation tasks is crucial for a practical unlearning method.\n- Our method is specifically designed to minimize changes to the overall model by only updating the text encoder parameters related to the target concept. Theoretical analysis (*e.g.*, Section 3.2, Lemma 3.8, and Proposition 3.11) shows that if the encoder output maintains uniformity on the hypersphere, unrelated prompts are largely unaffected.\n- In the main paper, we show that related but non-target concepts (*e.g.*, \"children at park,\" \"male person\") retain their generative quality after unlearning (see Table 1, Figure 2, Figure 3). Also in the appendix, Table 4 reports near-identical FID scores (considering the randomness in image generation) for unrelated concepts (*e.g.*, \"children with balloons\" and \"policemen with guns\" vs. the target \"children with guns\").\n- In response to the review, we have run further experiments. We now benchmark 50 diverse MS-COCO prompts (*e.g.*, \"a dog catching a frisbee\") to check whether Clipout degrades the model's overall generative quality on a broader prompt set. \nGiven a prompt, the score is calculated between the images generated by the original encoder and the images generated by the unlearned encoder. All images are randomly generated. The average score is reported in the table below:\n| Model    | Average FID Score |\n| -------- | ------- |\n| Original Model  | 151.03    |\n| Model w/ \"children with guns\" Unlearned | 157.79 (+4.48%)    |\n| Model w/ \"a naked woman\" Unlearned    | 163.52 (+8.27%)  |\nwhere the Original Model denotes the results between two different random seeds.\n\nSince the unlearning process aims to change the model parameters and the prompts may already contain or overlap with words in the target prompts, it is inevitable to see minor performance drops. Compared to the FID score change with respect to the target prompt (*e.g.*, Table 4), \nextensive new experiments confirm that the overall generative ability of the model on unrelated prompts is largely preserved.\nWe will add a table summarizing these broad-prompt evaluations in the revised version or appendix. We thank the reviewer for highlighting this important validation step.\n\n**Q2:** The paper proposes to use random clipping of the embedding of the target concept. What about other methods, such as perturbing the target embedding with Gaussian noise at different levels?\n\n**A2:** We appreciate the suggestion to compare random masking (as in Clipout) with other disruption strategies, such as adding Gaussian noise to the target embedding. Our choice of random masking is motivated by several considerations:\n- Random masking (Clipout) is closely related to dropout, which has strong theoretical guarantees for increasing representation diversity and reducing overfitting [1]. In our context, it directly destroys the semantic continuity of the target embedding, making it hard for the model to recognize or reconstruct the target concept.\n- Our negative-only contrastive loss is designed to maximize the distance between different masked versions of the same embedding. Gaussian noise, while also perturbing the embedding, does not guarantee that the resultant vectors will be sufficiently dissimilar in a structured way. Masking creates more pronounced and interpretable disruptions.\n- Gaussian perturbation with small variance is often insufficient to break the concept, as the noisy embedding may still be close in the latent space. With large variance, the embedding may be driven far from the data manifold, leading to instability or unpredictable side effects on model behavior. Clipout maintains a balance: it is strong enough to disrupt the target concept, but structured enough to avoid drastic off-manifold effects that may degrade unrelated generations.\n\n**Q3:** Current research shows that unlearning methods are often vulnerable to adversarial attacks. Could the authors analyze the robustness of the proposed method?\n\n**A3:** We acknowledge the emerging literature showing that machine unlearning methods, especially in generative models, may be circumvented by adversarial prompts [2]. Robustness is an important consideration for any unlearning method.\n- Most adversarial attacks attempt to recover the \"unlearned\" concept by either: a) Finding prompts semantically close to the erased concept (prompt engineering). b) Exploiting residual information in the model parameters and forging adversarial prompts. \n\n- Our method specifically destroys the alignment of the target embedding in the text encoder space. As demonstrated in Table 5 in the appendix, after unlearning, even synonyms or paraphrases of the target prompt are no longer mapped to the same region in the embedding space, indicating reduced susceptibility to prompt-based adversarial attacks.\n\n- For the theoretical basis, Proposition 3.11 ensures that unlearned embeddings converge to a uniform distribution. Re-activating the concept would require reversing this irreversible dispersion (Lemma 3.10).\n\n**Ref:**\n\n[1] Srivastava, Nitish, et al. \"Dropout: a simple way to prevent neural networks from overfitting.\" The journal of machine learning research 15.1 (2014): 1929-1958.\n\n[2] Zhang, Yimeng, et al. \"To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024."}, {"review_id": "BaeJH78tcP", "reviewer": "Reviewer_GHij", "summary": "Clipout introduces a negative-only contrastive unlearning strategy for a text-to-image diffusion model by editing only its text encoder rather than the full generator. Specifically, it takes the text embedding of a target prompt (e.g., “children with guns”) and create randomly masked variants of the embedding.  Then, the encoder is updated with a contrastive loss that pushes the masked variants away each other so that the original concept loses a coherent representation, while leaving other concepts untouched, requiring only these self-generated negative samples.  Theoretical analysis links the objective to maximising alignment loss for the target embedding while preserving the uniformity of the overall feature distribution. From experiments, Clipout achieves larger drops in CLIP similarity and identity-matching scores for the target while reducing impact on unrelated concepts compared to baselines on three scenarios: (1) personalized faces learned via Textual Inversion, DreamBooth, or LoRA; (2) built-in harmful prompts; and (3) copyrighted style prompts such as famous artistic styles.", "questions": "Questions are included in Strengths And Weaknesses", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. This work proposes a simple contrastive-learning technique that uses only negative samples and shows that it can erase a target concept without relying on any surrogate concepts.\n\n2. The authors take a novel perspective by analysing concept erasure through the lens of contrastive learning’s two core propeties: alignment for removal of the target concept and uniformity for preserving all the others with theoretical analysis.\n\n**Weaknesses**\n\n1. One of reviewer’s major concerns is the lack of important baselines in this work. For erasing target concepts while preserving the remaining ones, this paper does not compare against several key methods in this field, such as SA [1], UCE [2], MACE [3], SPM [4], and CPE [5]. Many critical components addressed by these prior works for successful unlearning in text-to-image diffusion models are largely ignored. While the proposed method erases concepts without relying on anchor or surrogate concepts, it is essential to compare against these baselines to clearly demonstrate its strengths on performance.\n\n2. Another concern is the lack of comprehensive benchmarks. The experiments are conducted only on object/celebrity erasure with prompts that explicitly contain target concepts. However, recent concept removal approaches emphasize the importance of removing implicitly contained target concepts within prompts, since failure to address this allows for bypassing of the concept erasure. It is unclear whether the proposed prompt-specific method can effectively remove such implicit content. To address this, experiments on I2P benchmakr—a widely used benchmark for evaluating explicit content erasure adopted by many baselines (UCE [2], MACE [3], CPE [5], RECE [6])—are necessary.\n\n3. Recent works on concept removal have identified three essential criteria for successful erasure: efficacy (effectively removing target concepts), specificity/locality (preserving unrelated content), and robustness (resistance to prompt attacks that regenerate erased content). However, this paper does not address robustness, which is a critical issue in this field. Red-teaming tools like RAB [7] and UnlearniDiffAttack [8] have shown that several concept erasure methods are vulnerable to adversarial prompts. Recent methods like CPE, RECE, and AdvUnlearn [9] emphasize the importance of defending against such attacks. The absence of any robustness evaluation in this work is therefore a serious limitation.\n\n4. It also does not examine performance in multi-concept erasure scenarios, as done in UCE, MACE, and CPE. The paper should demonstrate how well the method handles the removal of multiple concepts simultaneously.\n\n5. Despite presenting several theoretical analyses, the paper lacks a theoretical explanation for how preserving uniformity contributes to preserving unrelated concepts. This connection should be theoretically or empirically justified in more rigorous manner.\n\n6. While the authors claim that their fine-tuning process is faster, other efficient baselines have already been proposed after ESD and ASD as training-efficient and even training-free approaches. The paper should include comparisons of computational and memory efficiency with these recent baselines.\n\n**References**\n\n[1] Heng et al. \"Selective amnesia: A continual learning approach to forgetting in deep generative models.\" Advances in Neural Information Processing Systems 36, 2024\n\n[2] Gandikota et al. \"Unified concept editing in diffusion models.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.\n\n[3] Lu et al. \"Mace: Mass concept erasure in diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.\n\n[4] Lyu et al. \"One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.\n\n[5] Lee et al. \"Concept pinpoint eraser for text-to-image diffusion models via residual attention gate.\" The Thirteenth International Conference on Learning Representations, 2025.\n\n[6] Gong et al. \"Reliable and efficient concept erasure of text-to-image diffusion models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[7] Tsai et al. \"Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models?.\" The Twelfth International Conference on Learning Representations, 2024\n\n[8] Zhang et al. \"To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2024.\n\n[9] Zhang et al. \"Defensive unlearning with adversarial training for robust concept erasure in diffusion models.\" Advances in neural information processing systems, 2024.", "was_revised": true, "final_justification": "The authors partially addressed the reviewer's concerns, but this submission still raises limitation in terms of lack of recent baselines and  robustness. Therefore, the reviewer will maintain the score.", "rebuttal": "Thank you for your review and thoughtful comments. We appreciate your positive feedback acknowledging the novelty, conciseness, and theoretical basis of our method. We apologize for the issues in the preliminary manuscript and welcome the chance to clarify and improve the paper. Please see our point-by-point responses below.\n\n**Q1:** Lack of Important Baselines (e.g., SA, UCE, etc.).\n\n**A1:** We appreciate the reviewer highlighting the importance of comprehensive baseline comparisons. Our initial focus was on methods most widely adopted and publicly available at the time of writing, namely ASD [1], ESD [2], SLD [3], and FSMG [4]. However, we agree that a rigorous and fair evaluation requires including recent and specialized unlearning methods such as SA [5] and UCE [6], which have set new standards for both erasure quality and preservation of unrelated content in diffusion models. \n\nWe are actively running experiments to integrate these baselines into our evaluation pipeline, benchmarking Clipout against them on the same datasets and metrics, based on their public codes to ensure fairness. Following your suggestions, preliminary results on VGGFace2 are reported as follows:\n\n| Method    | FDFR (sks) ↑ | FDFR (female) ↓ | FDFR (Diff) ↑ | ISM (sks) ↓ | ISM (female) | ISM (Diff) ↓ |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| FSMG  |   0.76 (+0.70)  |  0.03 (+0.03)  |  0.73 (+0.67)  |  0.29 (-0.37)   |   0.22 (-0.06)  |  0.07 (-0.31)  |\n| ASD |   0.03 (-0.03)  |  0.03 (+0.03)  |  0.00 (-0.06)  |  0.55 (-0.11)  |  0.27 (-0.01)  |   0.28 (-0.10)  |\n| ESD    |  0.33 (+0.27)   |  0.01 (+0.01)  |  0.32 (+0.26)  |  0.24 (-0.42)  |  0.18 (-0.10)  |   0.06 (-0.32)  |\n| SLD    |  0.44 (+0.38)   |  0.02 (+0.02)  |  0.42 (+0.36)  |   0.31 (-0.35)  |  0.04 (-0.24)  |  0.27 (-0.11)  |\n| SA    |   0.21 (+0.15)   |   0.03 (+0.03)   |   0.18 (+0.12)    |   0.34 (-0.32)   |   0.23 (-0.05)   |   0.11 (-0.27)   |\n| UCE    |   0.27 (+0.21)   |   0.02 (+0.02)   |   0.25 (+0.19)   |   0.29 (-0.37)   |   0.19 (-0.09)   |   0.10 (-0.28)  |\n| Clipout (Ours)    |  0.94 (+0.88)   |  0.03 (+0.03) |  **0.91 (+0.85)**  |  0.33 (-0.33)  |  0.31 (+0.03)  |   **0.02 (-0.36)**  |\n\nIn the above table, values inside the bracket denote the difference compared with the results in the original pre-trained model.\nPreliminary results indicate that Clipout remains competitive, particularly in effectiveness, specificity, and preservation of non-target concepts. We appreciate this suggestion and will ensure these results with details are incorporated in the final paper.\n\n\n**Q2:** Lack of Comprehensive Benchmarks, Especially on I2P.\n\n**A2:** We appreciate the reviewer’s point regarding implicit concept erasure and the importance of evaluating on comprehensive benchmarks such as I2P. While our paper demonstrates strong results on explicit prompt erasure (object/celebrity), we agree that implicit concept removal is crucial for real-world safety.\n\nIn the appendix (see Table 3 and Section B), we have included initial results on the I2P benchmark, focusing on the “Sexual” category, where Clipout outperforms SLD in reducing unsafe generations. However, we acknowledge that a more thorough and systematic evaluation on the full I2P benchmark, as well as on prompts with implicit target concepts, is necessary. We are expanding our experiments to include these scenarios and will update the manuscript accordingly. We thank the reviewer for emphasizing this gap.\n\n**Q3:** Lack of Robustness Evaluation (Prompt Attacks, Red-Teaming, etc.)\n\n**A3:** We agree that robustness against adversarial prompts and red-teaming attacks is an essential criterion for practical concept erasure. Our current evaluation focuses on efficacy and specificity but does not directly address robustness to adversarial prompt engineering. \nBesides, our method ensures robustness by breaking the target concept's alignment in text encoder embeddings. Appendix Table 5 shows that our method can mitigate adversarial attacks based on synonym prompts. Theoretically, the irreversible uniform dispersion of unlearned embeddings (Proposition 3.11, Lemma 3.10) prevents concept reactivation.\nThis is a valuable direction, and we will also discuss more potential limitations and mitigation strategies in the manuscript.\n\n\n**Q4:** Multi-Concept Erasure Scenarios.\n\n**A4:** Thank you for pointing out the importance of multi-concept erasure. In the appendix (Section B), we mention that Clipout can be sequentially applied to multiple concepts, and initial qualitative results (see Line 592 and Figure 3) show that erasing one concept does not significantly affect others. However, we agree that a systematic quantitative evaluation, as performed in UCE, is necessary. We will discuss more about multi-concept erasure in revisions.\n\n\n**Q5:** Theoretical Connection Between Uniformity and Preservation of Unrelated Concepts.\n\n**A5:** We appreciate the request for a more rigorous justification of how preserving uniformity contributes to the preservation of unrelated concepts. In Section 3.2 and the appendix (Lemma 3.8, Proposition 3.11), we provide a theoretical basis: uniformity ensures that the feature space remains maximally informative, so that only the alignment (semantic similarity) for the target concept is disrupted while the global distribution of features is preserved. To further clarify, we will add both a more detailed theoretical discussion and additional empirical ablations (*e.g.*, measuring uniformity and downstream performance on unrelated prompts before and after unlearning) to reinforce this connection. Thank you for highlighting the need for a clearer exposition.\n\n**Q6:** Computational and Memory Efficiency Comparisons with Recent Efficient Baselines\n\n**A6:** We will benchmark recently proposed training-efficient and training-free methods for both runtime and GPU memory on our hardware, and add these results to Figure 5 and the efficiency section. Our preliminary implementation suggests our training-based method, Clipout, is still highly efficient, but we will be transparent about any cases where other methods are faster or more memory-efficient.\n\n\n**Ref:**\n\n[1] Kumari, Nupur, et al. \"Ablating concepts in text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Gandikota, Rohit, et al. \"Erasing concepts from diffusion models.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[3] Schramowski, Patrick, et al. \"Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Van Le, Thanh, et al. \"Anti-dreambooth: Protecting users from personalized text-to-image synthesis.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[5] Heng, Alvin, and Harold Soh. \"Selective amnesia: A continual learning approach to forgetting in deep generative models.\" Advances in Neural Information Processing Systems 36 (2023): 17170-17194.\n\n[6] Gandikota, Rohit, et al. \"Unified concept editing in diffusion models.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024."}, {"review_id": "M3pS6sYmqB", "reviewer": "Reviewer_NiNp", "summary": "This paper proposes a contrastive-based loss to fine-tune the text encoder or concept encoder. By masking some values of the concept embedding and constructing the positive and negative sample pairs, the supervision signals are obtained and used to optimize the concept encoder. Based on such a simple operation, the target concept can be removed from the pretrained diffusion models. The results show some effectiveness of this method with both quantitative and qualitative results.", "questions": "What's the intuitive idea of contrastive unlearning?", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n1) The contrastive unlearning method is quite simple and effective based on a complete theory analysis.\n2) This method can take into account both the erasing performance and the memory efficiency, and the baselines compared in experiments are sufficient.\n\nCons:\nThe most severe problem is that the whole writing is very poor, making the reading process very difficult. I will give a few examples here:\n    a) The abstract part lacks a high-level, intuitive explanation of the contrastive unlearning and indicates which parameters you are optimizing.\n    b) In Figure 1, what's the relation between the pretrained encoder in the training stage and the unlearned encoder in the inference stage? The unlearned encoder does not even have any explanations in this figure or the corresponding caption.\n    c) In Line 236 ``Personal concept'', this part you are indicating some new application with your technique, but later in Line 246, you start to introduce some implementation details.\nThese kinds of writing problems are everywhere in this paper, making it hard to read.", "was_revised": true, "final_justification": "Since the authors promise to polish the clarity of this paper, my main concern is partly addressed. I raised my score from 2 to 3.", "rebuttal": "Thank you very much for your careful review and detailed feedback. We are gratified by your favorable remarks regarding the simplicity, effectiveness, and theoretical basis of our method, as well as its performance-efficiency balance and the sufficient baselines used. We sincerely apologize for the issues with writing quality and clarity, and we appreciate the opportunity to clarify and improve the manuscript. Please find our point-by-point responses below.\n\n**Q1:** The abstract does not provide a high-level, intuitive explanation of \"contrastive unlearning\" or clarify which parameters are being optimized.\n\n**A1:** Thank you for pointing out this lack of clarity. The core intuition behind our method, \"contrastive unlearning\", is as follows:\n- Imagine you want a model to \"forget\" a specific concept (*e.g.*, \"children with guns\") so that it can no longer generate images matching that concept, but you want to avoid retraining from scratch or harming the model’s ability to generate other concepts. Our approach targets the text encoder (*e.g.*, the CLIP encoder used in Stable Diffusion models) that maps text prompts to embeddings, which are then used to condition the image generator.\n- Instead of using positive pairs to bring similar examples together as in standard contrastive learning, we do the opposite:\n    - We generate multiple \"masked\" versions of the embedding for the target concept by randomly zeroing out parts of its vector (think of this as creating noisy, incomplete versions of the same idea).\n    - We then train the encoder so that these different masked versions become dissimilar to each other, effectively scattering them apart in embedding space.\n    - This makes it impossible for the model to reliably map the target prompt to any specific, meaningful image, thereby \"removing\" the concept.\n- We only optimize the parameters of the text encoder. The rest of the model (*e.g.*, the U-Net and VAE architecture in Stable Diffusion models, *etc*.) remains fixed. This is a key design for efficiency and precision: we only unlearn the part of the model responsible for producing the concept embedding, leaving the rest of the system intact.\n\nTo sum up, Clipout removes harmful concepts (*e.g.*, copyrighted styles or private faces) from text-to-image models by optimizing only the text encoder's parameters. Inspired by contrastive learning, it uses only negative samples, which are generated by randomly masking units in the target concept's embedding, to push these variants apart in the feature space. This breaks the model’s ability to associate the prompt with the concept, without retraining the entire model.\nWe will revise the abstract to explicitly include this high-level intuition and clarify the optimization target.\n\n\n\n**Q2:** Figure 1 does not explain the relationship between the pretrained encoder (training) and the unlearned encoder (inference). The unlearned encoder is not explained in the figure or caption.\n\n**A2:** Thank you for highlighting this. The intention of Figure 1 is to illustrate the two main steps:\n- Starting from the pretrained encoder, we perform the \"Clipout\" unlearning procedure with the target prompt, generating masked embeddings and updating the encoder.\n- The result is an unlearned encoder (with modified parameters) that is then used at inference time. The rest of the system (the pretrained generator) is unchanged.\n\nDuring unlearning (*i.e.*, fine-tuning the encoder), the pretrained encoder is used to generate the embedding for the target concept, which is then randomly masked to create negative samples. The encoder is then updated (while the generator remains fixed) to maximize the dissimilarity between these masked variants. After unlearning, the updated (unlearned) encoder replaces the original encoder, so the model can no longer generate the removed concept.\nWe will update Figure 1 and its caption to clearly indicate this flow and explicitly label the transition from pretrained to unlearned encoder, as well as explain that only the encoder is updated.\n\n**Q3:** The writing structure is confusing, with implementation details mixed into application motivation, making the paper hard to follow.\n\n**A3:** We recognize that the organization of Section 4 (\"Evaluation\"), particularly the \"Personal Concept\" subsection, could be improved for clarity. Our intent was to first *motivate the practical need* for unlearning personal concepts (*e.g.*, deepfakes), and then to describe how we evaluate our method on these cases. Similar writing structures of the evaluation part were acknowledged by previous works [1, 2] in the related community. Following your suggestions, we will separate motivation and application discussions from implementation details in revisions.\n\n\n**Q4:** What is the intuitive idea behind contrastive unlearning?\n\n**A4:** Contrastive unlearning is the process of making a model \"forget\" a specific concept by making its internal representations for that concept unreliable and uninformative. In addition to the intuition explained in **A1**, we have the following points to complement:\n- Standard contrastive learning brings positive samples (*e.g.*, different views or augmentations of the same concept) closer together in embedding space, making them easier to recognize as the same.\n- Our proposed method exploits contrastive unlearning, which does the opposite for a target concept: a) Takes many corrupted (masked) versions of the target concept’s embedding. b) Fine-tunes the encoder so that these versions are as dissimilar as possible. c) As a result, the model loses the ability to consistently map the target prompt to a meaningful embedding, effectively \"forgetting\" how to generate that concept.\n- By only considering negative pairs (no positive anchor as in traditional contrastive learning), we ensure the model is not encouraged to \"remember\" any aspect of the target concept. This negative-only objective makes the forgetting process more robust and less likely to harm unrelated concepts.\n- Besides, the intuitive idea is backed up by theoretical claims:\n    - Alignment loss maximization (Eq. 10): Ensures variants of the target concept become dissimilar.\n    - Uniformity preservation (Lemma 3.8 and Proposition 3.11): Ensures unrelated concepts remain intact.\n    - Decoupling: Only the text encoder is unlearned (Sec. 3), avoiding full-model retraining.\n\n\n\n\n**Ref:**\n\n[1] Shan, Shawn, et al. \"Glaze: Protecting artists from style mimicry by Text-to-Image models.\" 32nd USENIX Security Symposium (USENIX Security 23). 2023.\n\n[2] Liang, Chumeng, et al. \"Adversarial example does good: preventing painting imitation from diffusion models via adversarial examples.\" 40th International Conference on Machine Learning, ICML 2023. 2023."}, {"review_id": "QJjToDeNPk", "reviewer": "Reviewer_49PU", "summary": "The authors introduce “Clipout” to unlearn private, copyrighted or inappropriate concepts in T2I VLMs, such as Stable Diffusion which include CLIP as a part of the diffusion model. The system is decomposed and unlearning occurs in a single part of the VLM: the encoder that produces the embedding to generate conditions for other parts in order to generate the concept. Clipout randomly “clips out” some units from the embedding and sets them to zero. Other parts remain fixed. The unlearning occurs for a particular concept (concept is described by the prompt in T2I generation) with a novel contrastive objective. The method is introduced with intensive theoretical analyses of the proposed methods.\n\nThe method proves successful in that the concept is erased leaving “related” concepts intact (i.e. can still be generated) and this is achieved with a reduced computational cost compared to other SOTA methods (the latter is demonstrated in the paper).", "questions": "1. How are the related concepts chosen and/or identified? Are these intended to be an exhaustive list? What is the potential for unintended ripple effects [1]?\n2. Given that the datasets are known to have limitations, i.e.. Celeb-A is questionably generalisable as it is based on North American celebrities,  and LAION-5b is known for offensive and problematic content that disproportionately impacts marginalised groups [2], how might this impact the generalisability of this work? Under which contexts is this likely to be effective, and are there any where it might not work as well?\n\n**References:**\n\n[1] Amara, Ibtihel, et al. \"EraseBench: Understanding The Ripple Effects of Concept Erasure Techniques.\" arXiv preprint arXiv:2501.09833 (2025).\n[2] Birhane, Abeba, Vinay Uday Prabhu, and Emmanuel Kahembwe. \"Multimodal datasets: misogyny, pornography, and malignant stereotypes.\" arXiv preprint arXiv:2110.01963 (2021).", "limitations": "The authors are commended for reflecting on their limitations. I recognise space is a consideration, but this is an important section and ideally shouldn’t be relegated to the appendix in my opinion. In addition to my questions above, I’d like to ask if the authors have reflected on the following potential biases stemming the use of models for evaluation (i.e. CLIP Score):\n\n1. CLIP is known to have biases [1-4], and given concerns of bias amplification [5], are these results likely to be limited in any way?\n\n**References:**\n\n[1] Hall, Siobhan Mackenzie, et al. \"Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution.\" Advances in Neural Information Processing Systems 36 (2023): 63687-63723.\n\n[2] Hong, Rachel, et al. \"Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp.\" Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 2024.\n\n[3] Agarwal, Sandhini, et al. \"Evaluating clip: towards characterization of broader capabilities and downstream implications.\" arXiv preprint arXiv:2108.02818 (2021).\n\n[4] Berg, Hugo, et al. \"A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.\" arXiv preprint arXiv:2203.11933 (2022).\n\n[5] Hall, Melissa, et al. \"A systematic study of bias amplification.\" arXiv preprint arXiv:2201.11706 (2022).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The authors are commended on presenting a well written paper that has extensive experiments to demonstrate a method of unlearning that can erase target concepts, while ensuring “related” concepts are intact in the generation process\n2. I believe this to be a timely topic to unlearn private, copyrighted, inaccurate and/or harmful content given rising concerns and discourse around these topics\n3. The localisation of the unlearning to the embeddings shows great potential and I am excited for this work to contribute to the related discourse\n4. The implementation is well documented and should be reproducible based on the descriptions\n5. I appreciate the author’s careful reflections on the implementation of these technical fixes “in the real world” (i.e. who decides what should be erased?) that balances the paper\n\n# Weaknesses\nI have outlined some concerns under “Questions” and “Limitations”.\n\n## Minor comments\n1. Please consider including this canonical reference in L27 for “harmful and offensive” content in large, web-scraped datasets: Birhane, Abeba, Vinay Uday Prabhu, and Emmanuel Kahembwe. \"Multimodal datasets: misogyny, pornography, and malignant stereotypes.\" arXiv preprint arXiv:2110.01963 (2021).\n2. The sentence describing what is meant by “concept” in L86-8 is potentially a few words (or should be reworded to make the meaning come across a bit more cleanly)", "was_revised": true, "final_justification": "The authors engaged well in the discussion and helped clarify any questions I had. My opinion is that this is a strong submission, and my original score reflected as such. Therefore, my score will remain 5:Accept.", "rebuttal": "Thank you sincerely for your careful review and insightful comments on our paper. We are particularly encouraged by your recognition of our ​​well-written presentation​​, comprehensive experiments, well-documented implementation​, and our careful ​​reflections on real-world implications​​. Our responses below aim to clarify ambiguities and directly address your concerns.\n\n**Q1:** Please consider including this canonical reference in L27 for “harmful and offensive” content in large, web-scraped datasets [1].\n\n**A1:** We have carefully reviewed the mentioned reference and we found it interesting and inspiring. We will include this in the revision. Thanks for your suggestions.\n\n\n**Q2:** The sentence describing what is meant by “concept” in L86-88 is potentially a few words (or should be reworded to make the meaning come across a bit more cleanly).\n\n**A2:** Thank you for highlighting this point. We agree that the definition of “concept” could be clarified for better readability. In the current text: *\"the term concept in this paper refers to a distribution that samples w.r.t. the same concept share the same semantic meaning, and it is described by the prompt in text-to-image generation.\"*\nThe current text can be improved to align with the \"latent class\" hypothesis [2] and is operationalized via prompt-conditioned embeddings (Sec. 3). \n- In this paper, a \"concept\" refers to a semantic idea or category represented by a text prompt in text-to-image models. All samples (*i.e.*, generated images) conditioned on prompts describing the same concept are expected to share similar semantic content. For example, the concept \"a photo of a cat\" encompasses all images generated from prompts that describe a cat, regardless of specific wording.\n- From the representation learning perspective, a concept refers to a semantic distribution in the embedding space, where samples belonging to the same concept share identical underlying semantics (*e.g.*, \"Van Gogh style\" or \"a specific person’s face\"), and this semantics is triggered by a text prompt during generation.\n\nWe will update the manuscript accordingly to ensure the meaning comes across more clearly and succinctly.\n\n\n**Q3:** How are the related concepts chosen and/or identified? Are these intended to be an exhaustive list? What is the potential for unintended ripple effects?\n\n**A3:** We would like to address the concerns about the selection of related concepts and the ripple effects as follows:\n- Related concepts are selected based on semantic proximity to the target concept, intuitively by modifying or substituting parts of the original prompt with similar or overlapping terms. For example, when unlearning \"children with guns\", related prompts might include \"children at park\", \"children with balloons\", or \"guns\". These are chosen to test whether the unlearning process affects only the intended concept or also impacts semantically close but benign concepts. For built-in concepts (*e.g.*, art styles), styles from distinct artists (*e.g.*, Picasso vs. Monet) are used to test specificity.\n- The list is not exhaustive, but is designed to cover high-risk cases (*e.g.*, Table 1, Figure 3). We select a representative set of related concepts for empirical evaluation, focusing on those most likely to exhibit potential side effects due to overlap in the embedding space. In practice, the set of all semantically related concepts is context-dependent.\n- There is always some risk of ripple effects, where unintentional modification of concepts that overlap in embedding space with the target may occur. Alignment and uniformity theory (Lemma 3.8, Proposition 3.11) ensures that unlearning only disperses the target concept’s embeddings, preserving others with numerical results supported (*e.g.*, Table 2). Our experiments (see Table 1 and Figure 3) show that Clipout minimizes such effects compared to prior methods. For example, it shows unrelated concepts (*e.g.*, \"a naked mole rat\") remain intact after unlearning the target (*e.g.*, \"a naked woman\").\n\n**Q4:** Given that the datasets are known to have limitations, i.e., Celeb-A is questionably generalisable as it is based on North American celebrities, and LAION-5b is known for offensive and problematic content that disproportionately impacts marginalised groups, how might this impact the generalisability of this work? Under which contexts is this likely to be effective, and are there any where it might not work as well?\n\n**A4:** We fully acknowledge the limitations of both CelebA-HQ and LAION-5B. CelebA-HQ is not globally representative, and LAION-5B contains noisy, potentially biased, and offensive content. These limitations have several implications:\n- Our empirical results are most directly applicable to the embedding distributions, not raw data. The effectiveness of Clipout in removing concepts is demonstrated on these benchmarks. Also, tests on VGGFace2 and I2P (offensive prompts) show consistent unlearning performance.\n- Clipout is likely to be most effective in contexts where the target and related concepts are well-represented and well-separated in the embedding space.\n- We will discuss these points in the Broader Impact and Limitations sections, and further clarify these caveats in the revised manuscript. Besides, we will reorganize some parts from the appendix to the main text.\n\n\n**Q5:** CLIP is known to have biases, and given concerns of bias amplification, are these results likely to be limited in any way?\n\n**A5:** We agree that CLIP’s biases are well-documented and may affect both the unlearning process and the evaluation metrics (*e.g.*, CLIP Score). Intuitively, since Clipout operates by modifying the CLIP-based text encoder, any pre-existing biases in CLIP (*e.g.*, gender, racial, or cultural biases) may influence which concepts are more or less easily unlearned, as well as the potential for unintended side effects. However:\n- Clipout only perturbs the target concept’s embeddings (Eq. 5), without retraining CLIP on biased data. Table 4 shows FID scores for related concepts (*e.g.*, children with balloons) change minimally.\n- Metrics like CLIP score difference (Table 1) explicitly measure unintended changes (*e.g.*, male vs. sks). For naked woman unlearning, naked mole rat and other concepts (Figure 3) remain unaffected, suggesting no systemic bias shift.\n- We also use ArcFace (not CLIP) for identity metrics (FDFR/ISM) to avoid compounding biases in experiments.\nWe will explicitly mention the potential for bias amplification and propagation in the revised manuscript, and cite relevant references [3, 4, 5, 6, 7] as suggested.\n\n\n**Ref:**\n\n[1] Birhane, Abeba, Vinay Uday Prabhu, and Emmanuel Kahembwe. \"Multimodal datasets: misogyny, pornography, and malignant stereotypes.\" arXiv preprint arXiv:2110.01963 (2021).\n\n[2] Wang, Tongzhou, and Phillip Isola. \"Understanding contrastive representation learning through alignment and uniformity on the hypersphere.\" International conference on machine learning. PMLR, 2020.\n\n[3] Hall, Siobhan Mackenzie, et al. \"Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution.\" Advances in Neural Information Processing Systems 36 (2023): 63687-63723.\n\n[4] Hong, Rachel, et al. \"Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp.\" Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 2024.\n\n[5] Agarwal, Sandhini, et al. \"Evaluating clip: towards characterization of broader capabilities and downstream implications.\" arXiv preprint arXiv:2108.02818 (2021).\n\n[6] Berg, Hugo, et al. \"A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.\" arXiv preprint arXiv:2203.11933 (2022).\n\n[7] Hall, Melissa, et al. \"A systematic study of bias amplification.\" arXiv preprint arXiv:2201.11706 (2022)."}]

Author rebuttal:

Meta-review: The paper proposes a novel approach to concept unlearning for text-to-image models based on a contrastive learning objective applied to clipped concept embeddings. 
All reviewers commend the novelty of the approach and acknowledge its effectiveness in erasing undesired concepts.

However, review scores for this paper were split, with multiple issues mentioned during reviews. Some reviewers mentioned the lack of comparisons to more recent concept unlearning approaches (while acknowledging that the work does compare to the canonical approaches). Another criticism was that the paper didn’t test the effect of the proposed approach on prompts that are completely unrelated to the unlearned concept, and didn’t test the robustness of the proposed approach to adversarial robustness. 

All these points are valid criticisms, but the authors addressed many of them during the rebuttal phase: they provided additional comparisons as requested demonstrating the competitiveness of their approach; they evaluated on unrelated prompts and tested robustness to adversarial perturbations. After rebuttal, the main outstanding criticism is the lack of comparison to two additional baselines mentioned by Reviewer GHij — but given that the paper already compares to 6 other methods, two of which were requested by Reviewer GHij and added during the rebuttal, I think that sufficient comparisons are provided to prove the merit of the proposed approach.

I note that Reviewer NiNp also voted for rejection, but on the sole basis of “poor writing” / “lack of readability”. Given that all three other reviewers seemed to have no problem understanding the technical content of the paper, and Reviewer 49PU even pointed out that the paper is well written, I will put low weight on this criticism.

In summary, I believe that the novelty of the proposed approach and the encouraging empirical results warrant acceptance. The authors have addressed many of the raised points and even if a few weaknesses remain, I think they are outweighed by the strengths of the submission. I thus recommend acceptance.
Decision: Accept (poster)