Zero-shot CLIP Class Forgetting via Text-image Space Adaptation
Authors: Alexey Kravets
Vinay P. Namboodiri
Venue: Accepted by TMLR ()

Abstract: Efficient class forgetting has attracted significant interest due to the high computational cost of retraining models from scratch whenever classes need to be forgotten. This need arises from data privacy regulations, the necessity to remove outdated information, and the possibility to enhance model robustness and security. 
In this paper we address class forgetting in vision-language CLIP model. Modern class forgetting methods for CLIP have demonstrated that zero-shot forgetting is achievable by generating synthetic data and fine-tuning both visual and textual encoders with a regularization loss. Our approach shows that class forgetting in CLIP can be accomplished in a zero-shot manner without any visual data by adapting the shared vision-text space of CLIP, thereby making the class forgetting process more efficient. Our method delivers superior results, demonstrating strong performance and complete class removal, regardless of the visual encoder used in CLIP. Furthermore, we explore what exactly is being targeted by the class forgetting algorithm discovering some interesting properties of CLIP features.

Published in Transactions on Machine Learning Research (01/2025)
Zero-shot CLIP Class Forgetting via Text-image Space Adaptation
Alexey Kravets Deportment of Computer Science University of Bath
ak3095 @both.ac.....oooo. uk
Vinay P. Namboodiri
vpn22@bath.ccm
Deportment Computer Science University of Bath
Reviewed on OpenReview: openrevvee... net/ net/forum? forum? id= V2SD2u VKEE
Abstract
Efficient class forgetting has attracted significant interest due to the high computational cost of retraining models from scratch whenever classes need to be forgotten. This need arises from data privacy regulations, the necessity to remove outdated information, and the possibility to enhance model robustness and security.
In this paper we address class forgetting vision-language CLIP model. Modern class forgetting methods for CLIP have demonstrated that zero-shot forgetting is achievable by generating synthetic data and fine-tuning both visual and textual encoders with regular- ization
loss. Our approach shows that class forgetting in CLIP can be accomplished in
zero-shot manner without any visual data by adapting the shared vision-text space of CLIP, thereby making the class forgetting process more efficient. Our method delivers superior results, demonstrating strong performance and complete class removal, regardless of the visual encoder used in CLIP. Furthermore, we explore what exactly being targeted by the class forgetting algorithm discovering some interesting properties of CLIP features. Full implementation can be found here!
1 Introduction
Class removal involves removing specific learned representations (or concepts) from a trained model without full retraining. This process aims to alter the model's understanding of a particular concept while retaining overall knowledge. In this work, we focus on class removal from the original CLIP model by OpenAI [Radford et al. 2021p that uses either ResNet50 ViT-B/16 visual encoders, where aim remove the associations between certain visual and textual representations in controlled manner. CLIP is vision-language model widely used in applications like robotics control (Shridhar et al. 2021), zero-shot object tracking (Solawetz 2021), and content moderation (Ahmed et al. 2023). As such, it is critical to ensure that CLIPP can forget specific concepts when proprietary or sensitive information is involved. For example, if private facial images inadvertently leak into the model during training, CLIP may recognize individuals, raising concerns under GDPR regulations. Such unintended associations could cascade across various applications, posing significant ethical and legal risks.
Class removal can be seen from the perspective of machine unlearning Xu et al. 2023 which involves removing specific data points from trained model related to the class to be removed. However, doing this for CLIP is challenging for the following reasons: (a) we do not have access to the original data of the class we want to forget that was used for training CLIP. Thus, any retraining of CLIP to achieve forgetting is not feasible. (b) CLIP a large parameter model. Even if we did obtain access the
Weights from httpa///////o/oo///////////////a
/CT.PP

---

Published in Transactions on Machine Learning Research (01/2025)
data that need to be forgotten from CLIP, fine-tuning CLIP would be challenging. To the best of our knowledge, only one work (Kravets& & Namboodiri 2025) has addressed zero-shot unlearning in CLIP. This study demonstrates unlearning in a zero-shot manner without requiring any real data. They indicate that changing weights in both the visual and textual encoders is nocessary for class forgetting. In contrast, we show that class lorgetting can be achieved without any synthelic or real data, by approaching the problem from the perspective of concept editing where we are not trying to remove the influence of training data but rather break certain associations in the model. Our method modifies only a small part of the textual encoder responsible for projecting the textual representation of the class into the shared image-text embedding space and does not require any synthetic data generation which can be time-consuming making class forgetting process relatively slow.
We recognize that at its core, the contrastive learning for CLIP aims to obtain a joint embedding space for the image and textual representation. Hence, we explicitly use projection of the textual representation to achieve forgetting. Our approach uses a direct optimization of a loss function to modify the text representation projection matrix. While doing so, we need to ascertain the gap between image and text representalion is modified only for a select set of classes that we desire to be forgotten while maintaining the gap between image and text representations for the classes that need to be retained. Once we do this for the text reprosentations, we observe that we achieve forgetting for the image-text classes that need to be forgotten and preserve the image-text correspondence for the other classes. After the optimization process is completed, an image for a retained class would still be close to the corresponding text representation. However, for the class that is forgotten, the image representation would be the same as the initial representation but the textual representation would be different. it has been explicitly modified for this class. For optimization we apply adaptation technique, low-rank adaptation (LoRA) (Hu et al. 2022), to find the minimum change in the text projection matrix optimizing a loss function that ensures that the change is such that the representation of the non-forget classes is retained while altering the representation of the forget class.
We do a performance comparison in Tab. 1 showing that our method both outperforms the previous meth- ods and is more robust. to different visual encoders achieving perfect class forgelting with ViT (Dosovilskiy lt al. 2020) and ResNet (He et al., 2015). We analyse through ablations the importance of the retain and forget loss components in Section 7.2 and how forget class projection place in the image-text space affects the forgetting ability of the model in Section 7.5) We find that retaining the knowledge of non-forget classes requires the inclusion of semantically similar classes, which can be generated using a large language model (LLM). This is because projecting the forget class to a different space primarily affects the closest classes in the image-text embodding space, thus, it is important to preserve this part of the space, while non- semantiialll similar classes are relained without explicit inclusion. We conduct a thorough ablation analysis on how the number of semantically similar classes affects performance in Section 7.3 Additionally, in Section 7.4 we assess how including semantically different classes affects performance.
We investigate how forgetting occurs. In Section 6.4 we show that there exist some "magic" neurons that the forgetting algorithm targets. These weights are such that changing them decrcases the dot product for the class to forget the most leading to a change in the class prediction. Furthermore, in the Appendix we show that there is a positive relation between the difficulty of forgetting a class and the Frobenious norm in the matrix of weights change. An overview of our approach is provided in Fig Our contributions are summarized as follows:
We improve current state-of-the-art CLIP zero-shot class forgetting as shown in Section 6.1
Forgetting is achieved without generating synthetic visual data improving efficiency. We show that no visual data are required to forget class and textual data are sufficient (Section
We provide a thorough analysis to understand our method. We show that there exist some "magic" neurons that our method targets to achieve forgetting and why it does that in Section 6.4 We also show that the Frobenius norm correlates with the difficulty of forgetting in the Appendix D
A detailed analysis provided in Section validates the choices made in our method and the gener- alizability of our method.
2

---

Published in Transactions on Machine Learning Research (01/2025)
-POOL AglK(P AP1)(()) AdlidP 1
Text Encoder
Opemazz change given by &P AD minimizing moving the lestioal rspresentation towards token representation.
from
Text Projection IP)
Image
AB?
ScaefardCare
SearfuudCar
DutordFlower
StartaadDeggs
Octondl SAMES
Tunable
Figure Overview of the approach. We utilize LoRA adapt the projection matrix of the textual representation into the shared image-text space. We ensure the representation for the forget classes is retained while altering it for the class to forget. To maintain the representation of the forget classes generate semantically similar classes using LLM. The forget class is projected into the empty token representation in the image-text space, In the figure we illustrate the forgetting for the Opel class.
2 Related Work
Multi-modal Concept Forgetting Li et al. (2024a) propose method to forget visual recognition of concopts using single image for multi-modal models. They begin by creating a multifaccted fino-tuning dataset aimed at aligning the forget concept with unseen concepts, assigning it a new visual description, decoupling factual knowledge about it, and preserving unrelated knowledge. The model then fine-tuned with this data using a dual masked KL-divergence loss. Similarly, Cheng & Amiri (2025) achieve forgetting with three-term loss function designed to ensure modality decoupling, uni-modal knowledge retention, and multi-modal knowledge retention. Authors of SalUn (Fan et al. 2024) propose to unlearn classification ion and generation models by computing the weights saliency and updating parameters based on it. These methods require real training examples and are not applicable to the CLIP dual encoder model.
In contrast, Zhang al. (2024) and Gandikota et al. 20033 achieve class forgetting the diffusion model by modifying cross-attention mcchanisms, thereby disrupting the associations between visual and textual representations of the concepts to be forgotten. Similarly, Gandikota et al. (2024) act on cross-altention in the diffusion model optimizing the key and value matrices mapping target concepts to preserving some other concepts. This technique allows editing, debiasing and crasure of concepts in the diffusion model. Since CLIP does not utilize cross-attention, these methods are not directly applicable.
one while
To the best our knowledge, Kravets & Namboodir] (2025) are the first to address zero-shot forgetting in CLIP by applying Lipschitz regularization. This approach guides the embedding of the visual and textual representations of the forget class towards perturbed embedding, breaking the visual-textual association while retaining knowledge of other classes. They achieve forgetting in zero-shot manner by generating
3

---

Published in Transactions on Machine Learning Research (01/2025)
synthetic visual data, thus climinating the need for real examples. The method involves modifying weights in both visual and textual encoders. In contrast, we show that simply updating the projection matrix from the text to image space is sufficient to achieve forgetting still in zero-shot manner without requiring any real or synthetic images and forgetting more efficiently as synthetic image generation requires time.
Model Adaptation Model adapters are task-specific modules added to a pre-trained model to enable it to efficiently adapt to new downstream task without retraining the entire model. Houlsby el al. (2019) inserted sequentially a small multilayer perceptron (MLP) layer between the layers of pre-trained BERT
model while freezing the original pre-trained paramcters. Similarly, Chen et al. (2022) added MLP layers but in parallel to the original frozen MLP connecting them in residual fashion showing superiority compared
to the sequential adapiation. Pfeiffer et. al. (202T) proposed a unified framework for training and sharing adapters across various tasks. Hu et al. (2022) introduced Low-Rank Adaptation (LoRA) technique which injects trainable low-rank matrices to learn task-specific information without altering the original pre-trained weights significantly. We utilize adapters in a different context, and specifically use LoRA to fine-tune the text projection matrix into the shared image-text embedding space in order to forget class in CL.IP.
3 Preliminaries
CLIP CLIP is a multi-modal model that understands both visual and textual inputs. It has been trained with a contrastive loss, which helps it learn to represent similar images and their textual descriptions closely in shared image-text embedding space while keeping dissimilar ones apart. Contrastive loss is defined as:
1 / Loss - 2 log Σ, exp(sim(2;, Uj))
N
exp(sim((r,, v1))
N log i=1
exp(sim(y,, sr,))
(1)
Σ, exp(sim (y:, x3,))
where sim(1;, yi) is the cosine similarity between an image and text embeddings and N the total number of image-text pairs. Contrastive training enables CLIP to perform various tasks, such as classification and retrieval, in a zero-shot manner.
CLIP employs dual encoder architecture with separate encoders for images and text. The zero-shot classification process in CLIP operates as follows: the input image is passed through CLIP's image encoder to generate image embedding T E Rd, that is also normalized, where d is the embedding size. Then, a set of textual descriptions corresponding to the target classes are generated, such as a photo of a felass). These textual class descriptions are converted into text embeddings by CLIP's text encoder, resulting in a classifier weight. matrix W Wrreproj E N where N represents the number of classes and the hidden dimension. Image and text embeddings must live in a shared image-text embedding space so that these can be compared against each other using the cosine similarity where the textual class corrosponding to the highest cosine similarity is the predicted class for given image. For this, W Wpreproj which lives in the text space is projected into the image space via a projection matrix P E R resulting in the final classifier weight matrix W = Wpreproj P E Nxd, which is also normalized. Mathematically, we have:
i = arg max TW? eEN
(2)
In our method for class removal, the projection matrix P plays crucial role in altering the textual repre- sentations to forget. specific classes.
LoRA LoRA is method that introduces trainable low-rank matrices into the model, allowing it to capture task-specific features without significantly modifying the original pre-trained model weights. This approach enables efficient fine-tuning by maintaining the majority of the model's learned knowledge while adapting to new lasks through a weigt set of parameters. LoRA assumes that the weight updates for a pre-trained model can be constrained to a low-rank subspace. Given a weight matrix Po E & in the model with & the input and d the output dimensions, LoRA modifies it as P = P. + AP where the change in the original Po matrix, AP, decomposed into a low-rank form AP = ABT with A E Rixr and B E dd being low-rank matrices with r < min(d, k). Only A and B are optimized while the original Po is kept frozen.
4

---

Published in Transactions on Machine Learning Research (01/2025)
4 Method
Loss Our method relies on an optimization approach that directly considers an explicit low-rank adaptation of the text projection matrix into the shared image-text representation space. The main principles we use are that the image-text representation space should be minimally changod for the classes that are to be retained and should be changed for the classes that are to be forgotten in a systematic manner such that forgetting is achieved. We also want the low-rank transformation matrices to be sparse to increase efficiency. These requirements lead to direct terms in our optimization approach. Note that similar optimization approaches for forgetting have been used previously for instance in LLM (Li et al. 2024b) and Stable Diffusion concept editing (Gandikota et al. 2024). However, our primary contribution does not lie specifically in the optimization approach. Instead, our main contribution is a straightforward method for achieving class lorgetting in CLIP by applying constraints to the lext projection matrix. This enables us to achieve forgetting in CLIP without requiring the actual data used for training (which is not available for CLIP). Further, our approach differs in being low-rank adaptation that provides an explicit forgetting to be achieved as the change in projection is known precisely through the low-rank projection adaptation. As we only change one matrix we can track what is being changed by our algorithm we find that there exist some "magic' neurons that the method targets in a specific manner to forget. This analysis is provided in Section 6.4 Also, we show in the Appendix that there is a positive relation between the difficulty of forgetting a class and the Frobenious norm in the matrix of weights change.
Given CLIP textual encoder fo that encodes input text into its vector representation and P the projection matrix that projects this representation into the image-text shared space, optimize the low-rank update to P given by the product of low-rank matrices A and B. We aim to minimize the following loss:
== A1 || + AP) (x,) (x,
A [[P + AP) fo(X,) F112+
As AP AP
where AP = ABT, X, represents the textual classes to relain, X3 represents lextual classes to forget and F is the new representation of the forget class that we discuss below. The first component of the loss ensures that classos to retain are maintained close to their original position in the embedding space. The second component modifies the projection matrix P such that the class to forget is projected into a new position of the image-text space F. The third component ensures that this done with minimum modification to P. We cannot include all classes seen by CLIP during its contrastive pre-training in X, since these are unknown. However, we find that including semantically similar classes to the forget class suffices to keep the repre- sentations of all the retain textual classes we tested on fairly untouched. Indeed, it is important to include semantically similar classes because when forgetting a class we perturb the space around that class which affects representation of similar classes, thus preserving those ensures that only the forget class is projected to a different part of the image-text space while retaining classes that were close to it in the embedding space. As we show in the ablations in Section 7.3 retaining any type of classes is not useful as reduces the performance on classes of the dataset the forget class was picked from. To generate semantically similar classes, we use large language model (LLM) using a prompt "Generate semandically similar classes to (class}'. These are shown in the Appendix.
we use the empty token representa- To determine where to project the forget class, denoted F in Eq. 3 tion. In the ablations in Section 7.5 we test. other variations such as a random projection and a perturbed representation of the forget class, which led to slightly worse results.
Determination of the Loss Parameters We fix a and Xa while X2 is is determined iteratively. At each iteration, we assess the reduction in the second component of the loss to evaluate whether the change in the projection matrix P is sufficient to project the forget class to the new chosen vector. We start from a fixed d and increment in small steps until the reduction in the second loss component exceeds 0.75% of its initial value. Additional implementation details described in the Appendix.
5

---

Published in Transactions on Machine Learning Research (01/2025)
5 Experiments
5.1
Comparable Methods
There exist only one directly comparable method on CLIP class forgetting, while other are adapted from other methods. We only compare our approach to zero-shot methods that do not require any real data.
Lipschitz CLIP Class Forgetting (Lip) To forget specific classes (Kravets & Namboodiri 2025) locally perturb both image and text representation of the forget class by Gaussian noise and minimize the Lipschitz regularization loss updating both the encoders. The method is zero-shot because, instead of the original images, synthetic images generated by gradient ascent are utilized.
Embedding regularization loss (Emb) Similar to the above, instead of Lipschitz regularization loss a simple difference between embeddings with L,2 regularization term used.
Amnesiac forgetting with synthetic data (Amns) The approach from Graves et al. (0221 h is adapted to multi-modal setting by fine-tuning CLIP using the same contrastive loss employed in its initial training, In this approach, the labels of the classes to forget are randomly replaced with different labels using synthetic data. To maintain zero-shot setting, data from the classes to retain are not utilized and solely data for the class to forget are employed to forget.
Error
Noise (EMMN) The approach from Chundawat et al. 2023 is adapted to multi-modal setting learning retain and forget samples through loss minimization and maximiza- tion respectively and training the model on these samples.
5.2 Datasets
Following (Kravets & Namboodiri 2025
we evaluate CLIP's forgetting capabilities four high-quality,
fine-grained datasets: Caltech101 Fei-Foii et al. 2007) contains images from 101 distinct categorics, cach
representing various objects or scenes. SlanfordCars (Krause el. al. 2013) contains images of cars of different makes and models. OxfordFlowers (Nilsback & Zisserman 2008) includes images of flowers of 102 different classes. StanfordDogs (Khosla et al.
2011 comprises 120 classes dogs of different species.
5.3 Evaluation
Ideally, to assess the forgetting procedure we should compare against the retrained model without the forget class. However, as CLIP training data are unknown and even if they were open sourced the computational against retrained model would be prohibitíve, we adopt similar logic to (Kravets
power required to & Namboodiri 2025 in order to assess how well the class has been forgotten. We want the accuracy the lorget class to be as low possible while maintaining the accuracy on other classes to similar level before forgetting. As we need to compare different quantities such as the drop in accuracy of the forget class, the remaining accuracy of the dataset the class was picked from and remaining accuracy on other datasets we create an aggregated metrics for an easier comparison. Given the normalized reduction in the accuracy of the class to forget Aa and normalized reduction the accuracy on the remaining classes for the N examined datasets, each denoted as A(ds): where one of those N datasets the dataset the forget class was picked from, we calculate the Average Score Kravets & Namboodiri (2225) metrics
Avg. Score = N + ((1 - ((1AA)) + Σ A(ds)).
(4)
ds
Best methods will have small average score. During evaluation we use the standard template A photo of a (class), however in the ablations we evaluate the forget. model with other templates to test the robustness to different evaluation templates.
It important to note that forgetting is a broad concept that cannot be easily guaranteed and there are multiple metrics available to evaluate it. We use forget accuracy (for classification) and precision@K (for
6

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Main forgetting results, classes for four selected datasets.
We compare our method to four other methods averaging across three
Method
Model
Dabsvtt
Avg: Target
Class mcc.
Classus Carr
Avg. Other
Avg.
Stus
Avg.
Caltsch101
Avg.
Avg. OxfordFlowwn
Avg. Score (4)
BF
AF
BF
AF
BF
AF
AF
BF
Ours
RNo0
StanfordCaars
0.397
0.0
0.358
0.55
0.517
0.51
0.857
0.855
0.061
0.837
0.007
Lip
RN30
StanfordCars
0.397
0.056
0.508
0.051
0.517
0.513
0.86
0.061
0.653
0:034
Kauh
RN50
StanfundCars
0.397
0.087
0.568
0.536
0.517
0.51
0.857
0.85
0.661
0.649
0.06
Amms
RN50
StanfordCarr
0.397
0:357
0.558
0.408
0.517
0.505
0:857
0.803
0.661
0.853
0:208
EMMN
RN30
StanfordCars
0,397
0.0
0.508
0.054
0.517
0.043
0.424
0.061
0.009
0.644
Ours
RN50
StanfurdDogs
0.693
0.0
0.516
0.509
10.558
0.554
0.857
0.866
0.661
0.658
0.007
Lip
RN50
StanfordDogs
0.093
0.048
0.516
0.516
0.508
0.558
0.857 0.866
0.661
0.666
0.018
Rech
RN50
StanfardDags
0.583
0.261
0.516
0.558
0.554
0.661
0.621
Amns
RN30
StanfordDogs
0.383
0.327
0.316
0.485
0.558
0.506
0.857
0.848
0.061
0.843
0.138
EMMN
RN50
StanfordDogs
0.593
0.0
0.516
0.053
0.558
0.107
0.807
0.493
0.661
0.107
0.304
Qurs
RN50
Calsech101
0.839
0.0
0.867
0.850
0.558
0.56
0.5117
0.513
0.661
0.058
0.002
Lip
RN30
Caltech101
0,839
0.081
0.857
0.865
0.558
0.357
0.517
0.52
0.061
0.657
010021
Emb
RN30
Caltech101
0.839
0.131
0.857
0.85
0.558
0.546
0.517
0.501
0.061
0.818
0.061
Amns
RN30
Calscch101
0.835
0.33
0.857
0.534
0.558
0.553
0.517
0.502
0.661
0.627
0.302
EMMN
RN5O
Collechlol
0.839
0.0
0.867
0.597
0.558
0.007
0.517
0.081
0.661
0.13
0.602
OowordFoowors
0.848
0:0
0.029
0.558
0:511
0.515
0.857
0.858
0.003
Lip
RN50
OxfeedFlowess
0.845
0.0
0.659
0.645
0.508
0.557
0.517
0.509
0.857
0.868
01.008
Baah
RN50
OxferdFlowrrs
0.848
0.442
0.669
0.625
0.558
0.553
0.517
0.5
0.857
0.85
0.122
Amus
RN30
OxterdFoowers
0.848
0.388
0.609
0.008
0.54
0.487
0.806
EMMN
RN30
Ozford
0.848
0.0
0.050
0.121
0.558
0:121
0.517
0.112
0.857
0.678
0.539
Qurs
VIT-B/16
StanfurdCars
0.695
0.0
0.666
0.642
0..901
0.591
0.933
0.984
0.708
0.703
0.006
Lip
VIT-R/16
StanfordCars
0.595
0.159
0.056
0.642
0.501
0.584
0:953
0.932
0:708
0.707
0.06
VIT-D/15
StanfordCars
0.595
0.0
0.626
0.557
0.391
0.508
0.921
0.708
0.09
Amms
VIT-B/16 StanfurdCars
0.695
0.345
0.666
0.18
0.501
0.398
0.933
0.876
0.708
0.51
0.327
EMMN
VIT-B/16
StanfordCars
0.595
0.159
0.656
0.182
0.591
0.119
0.933
0.589
0.708
0.137
0.002
Oure
VTT-/110
StanfondDog
0.673
0.0
0.501
0.582
0.0555
0.653
0.933
0.83
0.708
0.697
0.008
Lip
VIT-B/18
StanfordDogs
0.673
0.142
0.591
0.502
0.655
0.647
0:933
0.005
0.708
0.709
0.045
Emb
VIT-B/16
StanfordDogs
0.673
0.071
0.501
0.655
0.632
0.93
0.699
0.006
Amu
VTT-B/16 StanfundDoge
0.673
0.219
0.501
0.358
0.656
0.50
0.933
0.901
0.708
0.672
0.200
EMMN
VIT-B/16
StanfordDogs
0.673
0.042
0.591
0.385
0.055
0:284
0:933
0.826
0.708
0.438
0.501
Ours
VIT-B/16
Calsechl01
0.971
0.0
0.983
0.922
0.656
0.653
0.391
0.574
0.708
0.699
0.009
Lip
VIT: B/18
Calsechlo1
0.971
0.376
0,933
0.935
0.655
0.632
0.591
0.594
0.708
0.709
0.12
Eash
VIT-B/16
Callsch101
0.971
0.508
0.983
0.91
0.656
0.609
0.591
0.517
0.708
0.666
0.182
VIT-R/16
Calsecht01
0.971
0.846
0.055
0.517
0:581
0.145
0.708
EXMN
VIT D/18
Calsech101
0.971
0.284
0.903
0.813
0.635
0.352
0.591
0.302
0,708
0.473
0.341
VIT-B/16
OstosdFlownrs
0.784
0.0
0.707
0.705
00.666
0.654
00:501
0.584
0.993
0.923
0.004
Eip
WTT-//16
FlwwrdFoowwr
0.784
0.078
0.707
0.702
0.055
0.645
0.501
0.588
0.933
0.983
0.028
Emih
VIT-B/16 Fldord
0.784
0.0
0.707
0.617
0.656
0.545
0.591
0.522
0.953
0.906
0.089
Amns
VIT-B/16 OldordFlowers
0.784
0.834
0.707
0.527
0.655
0.602
0.501
0.526
0.933
0.913
0.307
EMMN
VIT-B/16
OxtesFlowers
0.784
0.02
0.707
0.433
0.655
0.317
0.391
0.304
0.933
0.83
0.305
retrieval) to measure the effectiveness of forgetting, However, other metrics such as Membership Inference Attacks (MIA) (Shokri et al. 2017) exist and offer different perspectives. In our case, we cannot apply MIA because it is typically used to check if specific training data remains embedded in the model. Since we do not have access to the training data used in CLIP's pro-training, applying MIA is not feasible in this context.
6 Results
6.1 Comparison against other forgetting methods
In [ we present the aggregated forgetting results across different methods with RN50 and ViT-B/16 B/16 visual encoders respectively. To compute them we first forget each of the 3 selected classes (can be found in Tab. 2 individually from each dataset and then average the results over these 3 classes. Granular results are found in the Appendix. Method column indicates the forgetting method used, Dataset column indicates the dataset from which the class to be forgotten was picked. The Aug. Torget Class acc. column denotes the accuracy on the target class averaged among the 3 selected classes when forgetting cach class individually while Aug. Other Classes acc. indicates the average accuracy on the remaining classes, excluding the selected forgotten class, also averaged while forgetting each class individually from the corresponding dataset. These results are shown before (BF) and after (AF) forgetting. The last eight columns represent the results on the remaining datasets reported both before and after forgetting. It's worth noting that Aug. Other Classes acc. measures the model's accuracy on the remaining classes from Dataset that are semantically similar
7

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Forgetting on multiple classes with RN50 and VIT-B/16 models.
Method Model Datases
Classes
Aug. Fanget Other
Classes Classes StanfordCare StanfordDoge Calrech101 OxfordFFowwes Arg. Score (1)
BF
BF
BF
AF
Up
R890
SankknlDiggr
Peltrowr fhr poosiiie Seotch berner
R:99
0.001
0.515
1.558
0.507
1.855
0.865
11001
0.035
Spilas 3500 Chese Cab.
2000 Dodze Rait
Li
R309
StankadCuss
21.1 Foed Runger SuperCki
01222
0.30
0.513
0.317
0.482
K1867
10.4
K1601
0
0.35
Lip
RN80
Calaci lot
suplecinns nitund plaiquue
6827
0.125
0.558
0 1.00
1.538
0.540
0.517
0.515
1.661
0.635
00:2
R850
Oofor FFwerss
giocis molkon
01996
Our
RN90
Fakiaws for paodla Seutch temrr
K:591
0.0
0.515
0 (90
1.535
0.51
1.853
0.851
11061
0.070
0.025
Speker Coupat
Rai
Cher Cab.
Oux
RNG0
Sank
2 Red Renger SuperChis
K300
0.0
0.56
0:55
0.517
0. (.000
1.857
0.55
X1661
0.634
0.021
Ouns
R890
Cahedi1o1
eepleroom. mitord platgau
11508
(10)
0.630
0.011
RNG0
Osfoc
0.016
VYT-8/16 Dang
Felenee poodlee Sootch better
10672
01.88)
11635
K1M33
015099
1.208
0.713
6.8
2009 Sinker CS Coupe
2010 Dodge Basz Piskpp 3500 Ches Cab.
VIT-B/16 StanderEluss
20.1 Runger Super
00,00
1.025
0.933
1.205
0
0.113
VIT-8/16 Caladis
eeplennun minurd plangure
0971
01.02)
1.500
1.655
11.90
0.200
C131
Li
VIT B-T-B-16 Oofor IF lwees
tompet coopar, graccii free milloww
0.31
11.105
0.68
1.655
(.580
11.00L
X.9333
0.929
0111
SankedDDe
Felinss Sootia
(1.00)
0.665
0.035
AgO Sye CS Crapa
2010 Dedor Ran Pakap 3514 Chon Cob.
VIT-8/16
Runger Superchis
0.600
1133
1.999
1.935
0.635
0.013
Orzs
VIT-B-11 Cabeek101
erplotou nihood platpee
0.0
[L002
11229
1.635
0.00
1.708
0.655
0.02
Oups
VIT-B/16 Usdee Floees
Lnmpee csapee wpw.ciia.iiioo.... fror mnllow.
8.507
0.0
0.7103
0 182
0.655
0.501
0.578
2.933
0.925
0.014
to the forget class (e.g. forgetting 'Pekinese' and testing on other breeds of dogs) while the final columns measure model's accuracy on semantically different classes. Tt's important to test both because, even if normally neighboring concepts of the forget class are the most affected by the forgetting procedure, sometimes semantically different classes might be affected as well. We show additional experiments in multi-label setting and with more diverse class distribution in the Appendix.
For both models, we observe that our method achieves superior performance in terms of the average score, balancing effective forgelting of the larget class while relaining information about the not largeted classes. Specifically, our method is able to remove completely the information about the forget classes from the model while other methods, apart from EMNN that however overforgets other classes, usnally retain some information. Lip is the most competitive with our method that sometimes achieves better accuracy the other classes retaining however some information about the forget class. Furthermore, comparing to Lip, which often struggles to forget well with a ViT visual encoder, our method is more robust and forgets well independently on the visual encoder used. Full results can be found in the Appendix. In the Appendix we also evaluate retrieval results and make some additional considerations.
6.2 Identity Forgetting
We validate our methodology on identity forgetting in line with the motivation of the right to be forgotten. We use PinsFaccs (Burak) dataset that contains 105 celebrity faces for this purpose. These results are presented in the Tab. 3 where we observe that our method works to forget different identities as well.
Table 3: Forgetting identities from PinsFaces da dataset.
Target
Other
StanfordCarrs StanfordDoge Caltuch101 OefordFlowers Avg. Score (-)
Model
Closs name Class acc. Classes acc.
BF
AF
BP
AF
BF
AP
BP
AF
BF
AF
BF
AF
RN30
Heury Caril
0886
0ul 082
1000
0.308
11508
1.517
0,312
1.857
0.961
0,001
11631
11,000
RN30
Amardy Crow:
0.838
0.0 0821
0813
0.358
0.550
0.517
0.514
0.857
0.858
0.061
0.654
0.005
RA30
Gal Giddt
(1707
0.1 08221
0.508
11222
1.517
0,536
1157
0.53
0.063
(1570
11.03
WT-B/16 16
Hnnry Chvil (0050)
0.0
0.97
0.02
0:653
0.617
0:591
0.574
0.033
0.005
0.708
0.707
0.01
VIT-B/10 16
Amnuai, Caew
0907
1.038
0.658
0.087
0.708
0.700
VIT-R/1 16
0.000
0.678
0.585
0.333
0.705
0.700
6.3 Forgetting on Multiple Classes
In Tab. 2 we show the results for Lip and Our methods when performing forgetting on multiple classes for RN50 and VIT-B/16 visual encoders respectively. Our method shows its superiority in terms of the average
8

---

Published in Transactions on Machine Learning Research (01/2025)
score also in this case. Again, our method is able to completely forget all the targeted classes while still maintaining high accuracy across not targeted classes while Lip, especially with the ViT-B/16 visual encoder retains substantial information on the classes to be forgotten. Indeed, Lip is less consistent across different architectures while our method is able to maintain this consistency in both single-class and multiplo-class lorgetting. We also show in the Appendix we that even if the forgetting requests for multiple classes arrive soquentially, our method performs well.
6.4 Understanding the Forgetting
Thanks to the simplicity of our method that only modifies one matrix we can closely examine what happens during the forgetting process. Specifically, we analyze which neurons in the projection matrix undergo the most significant changes following the forgetting procedure. This is done by looking at the absolute value of the AB matrix that represents the changes in the text projection matrix. Recall that all the projection matrix does is projecting the hidden textual representation into the text-image shared embedding space: from 512 into 1024 dimensions for CLIP with ResNet50 visual encoder and from 512 into 512 dimensions with ViT-B/16 16. We observe that there are some 'magic' neurons that the algorithm modifies more indicating that these textual features need to change the most to forget class while preserving the other classes. For example, for ResNet50 such neurons are in column 222 of the the weight projection change matrix (112, as can be seen in Fig. that shows on the x axis the column where most change occurred and on the y axis the sum of absolute values of the changes in that column.
Sum Values
Sum Values
4-
3-
1.
u.l. 11.1
222
400
600
800
2000
202
362
492
Neurons (Columns)
Neurons (Columns)
Figure 2: Sum of absolute values of neurons in different columns of the textual projection matrix change. On the left for RN50 visual encoder, on the right for VIT-B/16 16 visual encoder.
It turns out this is not random selection; plotting textual features across different classes and datasets, shown in Fig. g reveals that feature 222, which the forgetting algorithm targets, has the largest value. Thus, changing this feature is the easiest way for the model to forget a class, decreasing the dot product between the visual and textual features for that class. The corresponding visual feature 222 is also negative across images, so the network increases the value of the textual feature 222, decreasing the dot product for that class causing a change model's prediction. The presence of such ncurons and the algorithm's 's targeted modification intriguing. A similar phenomenon occurs with the ViT-B/I0 16 seen in the same figure. We provide more detailed example in the Appendix.
We do a similar analysis for Lipschitz forgetting, where tracking changes at different weight levels becomes challenging due lo the modification of many layers. However, we can observe the alterations in the final visual and textual features. Interestingly, when examining the features that change the most with Lipschitz forgetting, we observe the same pattern as with our method for the textual features. In contrast, this behavior not seen for the visual features where different features undergo more significant changes for different images.
9

---

Published in Transactions on Machine Learning Research (01/2025)
0.00
0.06
-001
0.65
Value
-0.03
Value
-3.04
-3.05
0.01
-0.06
0.00
-307-
222
400
600
800
1000
96
292
362
407
Features
Features
Figure 3: Textual features values averaged across different classes. On the left for RN50 visual encoder, on the right for ViT-B/16 visual encoder.
6.5 Forgetting is Specific
DT: toy poodla
OF: 2011 Ford Ranger SuperCab
The datasets in our main experiments include semantically similar classes that often share words. Our procedure effectively breaks the exact textual and visual association for the forget class but doesn't t break the association for similar words to the forget class as evidenced by the high accuracy on other classes from the same dataset that are often similar in meaning to the forget class. For instance, when removing the class information for toy poodle, classes miniature poodle and slandard poodle, which share the word poodle, maintain accuracy comparable to that before forgetting. To eliminate synonyms and similar words, the forgetting procedure would need to be repeated for those terms. We see this as a feature rather than a limitation, as it enables the preservation of much information as possible during the forgetting process. This approach provides precise control over which information to forget, including synonyms and similar words if necessary,
BF cuphorium
BF: bec mallow Ar: hbbacs
we show examples of the model's predictions before (BF) and after forgetting (AF). We observe that the new classes predicted by the model after forgetting are close to the correct ones indicating that our method targets specific knowledge of the model while proserving its general understanding.
In Fig. we
Figure 4: Predictions before (BF) and after forgetting (AF) with the prediction BF representing the tar- get class to forget.
7 Ablations
7.1 Variation of Templates for Evaluation
In these experiments we test how sensitive the model after forgetting is to the evaluation template and whether when changing it the model is still able to retrieve the forget class. Following (Kravets & Namboodiri 2025) we evaluate using the following three templates: "We can see [class} in this image", "This is a representation of {class}', "There is evidence of a {class} in the picture". Note that by changing the evaluation template also the accuracy of zero-shot CLIP changes. In Tab. we observe that forgetting is robust the change the evaluation template as the model is still unable to retrieve the forget class and maintains high accuracy of the not forget classes relatively to the model before forgetting.
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: Aggregated results across different evaluation templates. We aggregate across evaluations template to assess sensitivity of the models after forgetting to the change in the evaluation template.
Avg. Target
Avg.
Other
Avg.
Avg.
Avg.
Avg.
Class ace. Classes acc. StanfordCars Dogs Ca OxfordFlowees Avg. Score (U)
Model
Dataset
BF
BF
AF
AF
BF
AF
BF
AF
HF
AF
RN50
StanfordCars
0..272
0.0
0.193
0.488
0.415
0.415
0.81
0.811
0.519
0.518
0.00G
RN50
StgnfordDoogs
0.306
0.0
0.416
0.405
0.492
0.483
0.81
0.809
0.519
0.518
0.007
RN50
Calteh1011
0.879
0.029
0.81
0.81
0.492
0.488
3.415
0.415
0.519
0.517
0.01
RN30
OxfordFlower lowes
0.638
0.0
0,518
0.513
0.492
0.491
0.415
0.415
0.81
0.81
0.001
/91-
StanfordCars
00.497
0.621
0.638
0.516
0.514
0.88
0.882
0.61
0.61
0.003
VIT-B/10 B/16
StanfordDogs
0.532
0.516
0.504
0.622
0.617
0.88
0.88
0.61
0.607
0.008
VIT B/16
Caliehh101
0.97
0.879
0.88
0.621
0.516
0.513
0.61
0.61
0.004
VIT-B/16
OxfordFlowers
0.067
0.609
0.601
0.516
0.513
0.88
0.85
0.001
7.2 Loss Components Ablation
In this subsection we assess how important are the loss components to the forgetting procedure. For this, we first set Å to and then A3 to Results are shown in Tab. 3 where we observe that both components are important for forgetting that achieve the best results in terms of the average score when all the components are included. The average score drops more when X1 is excluded from the loss.
Table 5: Ablations the importance of the different components of the loss setting A to or Xg to
Method
Model
Avg. Target
Class
Avg. Other Classes
Avg. StanfordCars
Dogs
Avg.
Avg. Caltech101
Avg. OxfordFlowers
Avg. Score (4)
BF
AF
BF
AP
BF
AF
BF
AF
BF
AF
B3F
AF
All terms
RNG0
0.609
0.648
0.642
0.608
0.657
0.517
0.513
0.837
0.857
0.661
0.656
0.005
Excludinn A
RN50
0.660
0.01
0.648
0.625
0.558
0.558
0.517
0.511
0.857
0.863
0.661
0.644
0.017
Excluding bg RN50
0.609
0.004
0.648
0.641
0.608
0.553
0.517
0.509
0.837
0.855
0.661
0.655
0.03
All terms
VIT-B/16
0.756
0.722
0.715
10005
0.053
0.583
0:953
0.932
0.708
Excluding As
VTT B
0.756
0.185
0.722
0.694
0.658
0.653
0.591
0.587
0.933
0.935
0.708
0.702
0,061
Excluding As
VTT-B/10 /
0.756
0.001
0.722
0.711
0.005
0.636
0.091
0.578
0.933
0.928
0.708
0.094
0.019
7.3 Number of Classes to Retain
In this section we evaluate how varying the number of classes to preserve affects the forgetting results. In Tab. t we observe that reducing the number of classes to preserve the average score drops, but the results are still relatively robust even when only 10% classes to retain are used the most sensitive to the reduction in retain classes is the dataset the forget class was picked from. All the generated retain classos can be found in the Appendix, and on average we generate 100 semantically similar classes to the forget class for each dataset.
Table 6: Ablations on the number of classes to retain.
Number of Classes
Avg. Target
Avg. Other
Avg.
Avg. StanfordDoga
Avg-
Avg.
Score (L)
Model
Cless scc.
Classes acc.
StanfordCars
Caltech101
Oxford Flowers
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
B3F
AF
All 50%
RN50
0.669
0.0
0.648
0.642
0.558
0.566
0.517
0.509
0.857
0.856
0.661
0.635
0.007
RN50
0.038
0.648
0.636
0.558
0.504
0.51
0.57
0.661
0.633
0.01
10%
RN50
0.669
0.0
0.648
0.623
0.558
0.558
0.517
0.51
0.857
0.558
0.681
0.653
0.013
All
VIT-B/16
0.756
0.0
0.722
0.712
0.655
0.646
0.501
0.576
0.033
0.93
0.708
0.095
0.016
50%
VIT-B/16
0,756
0.0
0.722
0.704
0.655
0.65
0.591
0.576
0.583
0.931
0.708
0.098
0.015
10%
VIT-B/16
0.756
0.722
0.688
0.655
0.652
0.501
0.081
0.933
0.832
0.708
0.7
0.016
7.4 Retaining Classes from Forget Class Dataset
In our main experiments we used semantically similar classes (SemSim) to the forget class for the retain loss component. In Tab. 7 we compare the effects of using actual classes from the dataset the forget class picked from, denoted Clsy, and when using semantically different classes (SemDiff). Overall, we find
11

---

Published in Transactions on Machine Learning Research (01/2025)
that semantically similar classes are crucial for maintaining high Other Classes acc.. When forgetting, the original projection matrix is altered in a way that perturbs the space near the forget class more leading to a greater reduction in accuracy for semantically similar classes which are closer in the image-text embedding space compared to different classes, where the space is less affected. Using actual classes (Cls,) performs the best, but similarly to semantically similar classes generated by a large language model. In contrast, using semantically different classes, taken from the Food101 (Bossard et al., 2014) dataset, results in the worst outcome, especially for Other Classes acc. while the accuracy of the classes not semantically similar to the forget class (i.e. other test datasets) is maintained without explicitly including them.
Table 7: Ablations with actual (Cls,), semantically similar (SemSim) and different (SemDiff) classes
Typs of Retained
Classes
Model
Avg. Target
Class
Classes
Avg. Other
StanfordCars
Avg.
DtanfordDogs
Avg.
Caltech101
Avg.
Avg. OxfordFlowers Avg. Scarw (8)
BF
AF
BF
AF
DF
AF
BF
AF
BP
AF
BF
AF
RN50
0.618
0.558
0.558
0.517
0.51
0.857
0.661
0.859
0.004
SemSim
RN50
0.889
0.0
0.648
0.642
0.538
0.567
0.517
0.513
0.857
0.857
0.661
0.656
0.005
Sem
RN50
0.689
0,648
0.61
0.558
0.558
0.517
0.511
0.857
0.86
0.661
0,655
0.016
Cla.
VTT-B/16
0.756
0.0
0.722
0.718
0.655
0.852
0.591
0.583
0.933
0.931
0.708
0,702
0.006
SemSim
VIT B/16
0.756
0.722
0.715
0.635
0.653
0.591
0.583
0.933
0.932
0.708
0.7
0.008
SemDuB
VTT-D D/16
0.756
0.004
0.722
0.68
0.655
0.851
0.591
0.584
0.933
0.934
0.708
0.703
0.018
7.5 Forget Class Projection
We evaluate the importance of where to project the forget classes in the shared image-text space. We test different variations like projecting into a random vector and a perturbed embedding of the forget concepts comparing them to the emply token projection used in our main experiments. In Tab g we see that projection space is less important as similar results are achieved when we project into different parts of the space.
Table 8: Ablations on projection. We perform an ablation study projecting into the empty token vec- tor (Empty Token prof), random vector sampled from Gaussian distribution ( (Random proj) and perturbed embedding of the forget class (Perturbed proj)
Method
Model
Avg. Target
acc.
Classes acc.
Avg. Other
StanfordCars StanfordDogs
Avg.
Avg.
Caltech101
Avg.
Oxford
Avg.
Avg. Score (1)
BF
AF
BFF
AF'
B3F
AF
13P
AP
L3F
A.
LF
AF
EmppyyToben proj
RN50
0.609
0.0
0.648
0.642
0.558
0.567
0.517
0.513
0.857
0.857
0.661
0.656
0.005
Random peoj
RN50
0.669
0.0
0.648
0.642
0.308
0.567
0.517
0.511
0.857
0.806
0.661
0.656
0.006
Perturbed proj
RN50
0.689
0.0
0.648
0.642
0.358
0.557
0.317
0.511
0.857
0.858
0.661
0.635
0.006
EmptyToleen proj
WT B/16
0.756
0.0
0.7722
0.715
0.358
0.853
0.517
0.583
0.857
0.932
0.681
0.7
0.008
Random proj
VIT-B/16
0.756
0.0
0.722
0.713
0.558
0.662
0.517
0.58
0.857
0.931
0.681
0.608
0.01
Perturbed proj
VTT-B/16
0.756
0.0
0.722
0.713
0.558
0.602
0.517
0.58
0.867
0.931
0.661
0.608
0.01
8 Conclusions
In this work we demonstrated that it possible to forget a class the CLIP model without altering the original visual encoder, thereby eliminating the need to generate synthetic data. A learned adaptation to the projection matrix of the textual encoder, which projects textual representations into the image- text embedding space, is sufficient for class forgetting. Furthermore, semantically similar classes can be affected during forgetting, reducing their accuracy. Therefore, it is crucial to include semantically similar classes that are close in the embedding space in the loss function to retain their information.
show that the representation of
Acknowledgements
We'd like to gratefully acknowledge Microsoft's 's compute support through Microsoft's Accelerating Foundation Models Research grant and the support from University of Bath for the studentship.
12

---

Published in Transactions on Machine Learning Research (01/2025)
References
Syed Hammad Ahmed, Shengnan Hu, and Gita Sukthankar. The potential of vision-language models for content moderation of children's vidcos. In 2023 International Conference on Machine Learning and Ayplicalions (ICMLA), pp. 1237- 1241. IEEE, 2023.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101 mining discriminativv componttss with random forests. Computer Vision - ECCV 2014, pp. 446-461, 2014. doi: 10.10777977-3---111155594-----40
Burak. Pins face recognition dataset. URL htt.l//g.....aaaacccc////a//aaaa
S. Chen, C. GE, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (ods.); Advances Neural Information Processing Systems, volume 35, pp. 16664-16678, 2022.
Jiali Cheng and Hadi Amiri. Multidelete for multimodal machine unlearning. In European Conference on Computer Vision, pp. 165 184. Springer, 2025.
Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Trans. Info. Forensics and Security 18:2345 2354, 2023. doi: 10. 1111////1F..2000 2023. 3265506. URL lattps://ary //rxivv org/ attpaabb///..../ 05629
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CVPR, 10 2020.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via tradient-based weight saliency in both image classification and generation. In ICLR, 2024.
Li Fei- Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106:59-70, 04 2007. doi: 10.1016//..vin.2000599..
Rohit Gandikota, Joanna Materzynska, Jaden diffusion models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2426-24366 2023.
and David Bau. Erasing concepts from
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńskk, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5111-5120, 2024.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 12 2015. URL
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 2790-2200 27999, 09- 15 Jun 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https://opeerevieee net::forumidddndeWWeeeee.
image categorization. In First Workshop on http:://ppppeeeeeeeeeeee//mnnrnr Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fino-grained Visual Categorization, IEEE Conference on
13

---

Published in Transactions on Machine Learning Research (01/2025)
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cat- egorization. In Proceedings of the 2013 IEFE International Conference. on Computer Vision Workshops, ICCVW '13, pp. 554-561, 2013. ISBN 9781479930227.
A. Kravets and V. Namboodiri. Zero-shot class unlearning in clip with synthetie samples. In Winter Conference Applications of Computer Vision (WACV), 2025.
Jiaqi Li, Qianshan Wei, Chuanyi Zhang, Guilin Qi, Miaozeng Du, Yongrui Chen, Sheng Bi, and Fan Liu. Single image unlearning: Efficient machine unlcarning in multimodal large language models. In The Tighhh- Annual Conference Newral Information Processing Systems, 2024a.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, and Ann- Kathrin Dombrowski. The wmndp benchmark: Measuring and reducing malicious use with unlearning.
ICML, 2024b.
M. Nilsback and Andrew Zisserman. number of classes. 2008 Siath
Automated flower classification over a large Conference on Computer Vision, Graph-
Indian
ics Image Processing, 2008.
URL https://wwrt semanticscholar www.ccchoaar.rroopp.eoooo org/paper/
wwwngge-uubbbr--ilbbaaa... Zisserman/
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion Non-destructive task composition for transfer learning. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the Puropean Chapler of the Association for Computational Linguistics, pp. 487 503, April 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transfcrable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 8748 8763. PMLR, 18 24 Jul 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy (SP), 03 2017. URLhttpss pettps://ardas //arxiv.. htg/abb/1610..000 1610.05820001
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Solawetz. Object tracking with clip, 2021.
URL
https://blog.rroofooo.coo
zero- wwwpsshtttttttttrcrccceeeee
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip Yu. Machine unlearning: survey.
ACM Comput. Sure., 56(1), aug 2023. ISSN 0360-0300. doi: 10.1145/360362
URL https://doi. org/
10. 0111//603/3220
Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in texl-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 1755 1764, 2024.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Appendix
Table of Contents
RESNET FULL RESULTS
II VrT-B/16 FULL RESULTS
III. RETRIEVAL TASK
IV. INTERPRETING DIFFICULTY OF FORGETTING A CLASS
2
V. IMPLEMENTATION DETAILS
22
VI. SEQUENTIAL MULTI-CLASS FORGETTING
22
VII. DETAILED EXAMPLE ON MAGIC NEURONS
22
VIII. GENERATED SEMANTICALLY SIMILAR CLASSES TO PRESERVE
23
15

---

Published in Transactions on Machine Learning Research (01/2025)
A ResNet Full Results
Table 9: Forgetting results with RN50 visual encoder. We compare our methods with four others on three classes for four selected datasets.
Target
Other
Method Datasel
Class mame
Class acc. Classes acc.
StanfordCars Dogs Caltoch OxfordFlowers
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
IF
AF
Ours
Sogr
Pekinese
0.700
0.0
0..11
0.51
0..538
0.359
0.857
0.833
0.06]
0.650
Ours
Stan:foeDoge
Lay pooille
0.574
0.0
0.516
0.307
0.558
0.56
0.857
0.837
0.061
0.644
Ours
Sord
Seatch terrier
0.5
0.0
0.517
0.509
0.558
0.543
0.557
0.830
0.661
0.656
Ours
StarfordCars
2000 Spyker Coupe
0.262
0.0
0.359
0.56
0.517
0.509
0.857
0.858
0.661
0.658
Curs
Starfoddaas
2020 Dodge Ram Pickup 3500 Crew
0.405
0.D
0.358
0.542
0.517
0.512
0.857
0.851
0.061
0.654
Ours
Caars
2011 Ford Ranger SuperCai
01.524
0.0
0.558
0.549
0.517
0.508
0.857
0.856
0.061
0.658
Ours
Calsech 101
caphonium
0.780
0.0
0.858
0.861
0.558
0.561
0.517
0.512
0.661
0.66
Curs
1a
milnart
00.825
0.0
0.857
0.857
0.558
0.557
0.517
0.519
0.061
0.653
Ours
plalypus
0.9
0.0
0.857
0. 0.56
0.558
0.56
0.517
0.500
0.661
0.061
Ours
Oxford Flowers
gaazania
0.957
0.0
0.658
15588
0.555
0.517
0.514
0.857
0.850
Cure
Oxford Flowers
tree mallow
1.0
0.0
0.658
0.646
0.558
0.56
0.517
0.511
0. 0.857
0. 0.856
Curs
Oxfori
trunped crequer
0.588
0.D
0.661
0.561
0.558
0.56
0.517
0.516
0.857
0. 0.801
Lip
Star ford Dogs
Pokinuse:
0.705 0 066
00.515
0.514
0.655
0.559
0.983
0. 8.6
0.770
0.658
Lip
Soar Dogs
poodle
0.574
0,033
0.516
0.518
1.655
0.359
0.933
0
0,708
0.647
Lip
Doogs
Scotch beerier
0.5 0.017
00.517
0.516
0.655
0.357
0.933
0. 8.8
0.708
0.96
Lip
Starford SoleooCCans
2009 Spyler C8 Coupe
(1,262 0. 0.024
0.559
0.553
0.591
0.518
0. 0.983
0 8.666
01708
0.56
Láp
Star ford
2010 Dodge Ram Piekup 3500 Crew
0..40
0.143
00.558
0.544
0.591
0.502
0.993
0. 0.8415
0,770
0.698
Lip
StarfordCars
2011 Ford Ranger SuperCais
00.521 0.0
0..558
0.355
0.591
0.52
0.933
0. 1.800
0.708
0.66
Lip
Caltech101
euphonium
0.788
0.0
0. 0.858
868
0.655
0.557
0.501
1052
0.700
0.658
Lip
Calsech101
minaaee
0..82
0.013
0.857
0.863
0.055
0.336
0.590
11391
0.515
0,708
0.001
Lip
Cal
platypus
0.9 01.2
0:857
1655
0.358
0.591
0.524
0.708
0.653
Lip
OxfordFlomee
ggazania tree mallow
0.957 0.0
0 658
649
655
0. 355
0. 890
0 0.513
0.933
0. 8.000
Lip
Oxford FlfordFoowers
1.0 0.0
0.638
0.513
0.055
0.357
0.390
0331
0.933
0. 1.809
Lip
OxfordFloowes
Leumped creeper
01588
0.661
0.843
0.655
0.357
0.591
0.508
0.983
0.
Emb
ford Dogs
Pokinese
0.7705 0.361
10.515
0.484
0.558
0.559
0.857
10.661
0.633
Emb
Dogs
say poodle
0.574 0.361
0.516
0.508
0.553
0.857
0.
0.001
0.613
Emb
Sogg
Seotch teerier
0.5 0.002
0..17
0.472
0.558
0.351
0.857
0.837
0.001
0.617
Emb
2009 Sgylier C8 Caupe
(1,262 0.024
0.559
0.529
0.517
0.508
0.857
0.841
0.060
0.059 1585
Emb
StarfordCars
2010 Dodge Ram Pickop 3500 Crewe Cab
0.119
0..558
0.542
0.617
0.512
0.857
0. 8857
0,001
0.054
Emb
Starfordaas
2011 Food Ranger SuperCa's
(0.521 0.119
0.358
0.539
0.511
0.509
0.857
0.832
0.001
0.651
Emb
Caliedh101
cuploniium
(1,783
0.2633
0.858
0.833
0.558
0.548
0.5016
0.661
0.616
Emb
Caltorh 101
minure
00.806
0.13
0.857
0.827
1.588
0
0.517
0.507
0.061
0.630
Emb
10
platypus
0.9 0.0
0. 0.857
829
0.508
0.319
0.517
0:19
0.661
0.597
Emb
FlwwrrFsoweess
gasenia trre mallow
0.357 0.739
0.658
0.532
1.558
0.551
0.517
0.303
0.857
0.844
Emb
OxfordFlooeee
0.353
0.658
0:612
1.558
05554
0.517
0.504
0.857
0.
Emb
OxfordFlomees
trumpet creeper
00.588 0. 0.235
[0.00 0.661
632
0. 0.655
0. 0.356
0..517
0.008
0. 0.866
0. 8383
Armus
foed
Pekinuse
(0.705
10.515
1.558
0.857
0.
(0,061
0.65
Arms
Soar
say poodle
0.577 0 102
0.516
0 0-4
0.508
0.55
0. 0.867
0. 838
0.661
0.628
Anns
Dogg
Seotch beerier
0.5 0.0151
0..17
0.458
0.558
0.339
0.857
(1 0.839
0.001
0.652
Arnus
2009 Sgyloo Caupe
(1,262 0.143
0.550
0.516
0.517
0.51
0.857
0.
0,060
D.646
Armss
StarfordCars
2010 Dodge Ram Pickmp 3506 Crew Cab
0.429
(. 0..558
0.49
0.517
0.5
0 0.857
0. 888
0.661
0.658
Amns
StarfoedCass
2011 Feed Rouger SuperCab
(0.521 0.5
0.358
0.459
0.347
0.857
0.858
0.861 1
0.850
Amnu
Caltech111
cuphoniiu
(1.789
0.316
0,558
0.557
0.519
0.061
0.655
Arams
Caltoch 101
platypus
0.9 0.5
0 0.8
832
0.658
0. 55555
0.510
0. 495
0.661
0.634
Amms
Calsech161
milsszet
0.825 0.171
0.857
0.813
0.508
0.516
0
0. 193
0.661
0.591
Anus
Oxford Fonwwrss
gasanin
(1.57 0.87
0.658
0.585
0.558
0.357
4.9
0.857
0.834
Arans
Oxfordlllmeees
trme mallon
1.0 0.0
0 0.658
108
0.658
0-511
0.617
0. 0.476
0 0.857
0. 8.843
Arams
OxfordFloweee
trumpet creeper
0.088
0,294
0.661
0.384
0. 10S08
0.351
0.517
0.491
0.807
0. 828
EMMN
Starford Doge
Pekinese
U.787 0.0
0.59
0.376
0.655
02288
0.933
0.828
U.YUS
0.1322
EMMN
Soddoo
poodlee
0.607
0.0
0.501
0.373
0.308
0.933
0.833
0.708
0.446
EMMN
Star foed Dops
Seatch tervier
0.625
0.125
0.501
0.347
0.265
0.935
0.810
0.708
0.436
EMMN
StarfordCars
2009 Spyler C8 Coupe
0.129
0.0
0.656
0.188
0.591
0.116
0.933
0.611
0.708
0.148
EMMN
StarfordCars
2010 Dodge Ram Pickup 3500 Crew Cub
0.548
0.476
0.656
0.184
0.391
D.13
0.933
0.56
0.708
0.126
FMMN
Starfred lam
2011 Ford Rangee SaperCac
0.81
0.0
0.654
0.591
111
0.933
0.504
0.700
1.156
EMMN
Caltech101 101
euptonium
1.0
0.106
00:933
0. 0.783
0.655
0.352
0.591
0.297
0.708
0.15
EMMN
Calsech 101
minaret
0.913
0.348
( 0:933
0.817
0. 1.655
0.36
0.591
0.315
0.708
0. 0.485
EMMN
111
platypus
1.0
0:933
0.838
0.655
0.315
0.591
0.291
0.708
0.181
EMMN
Oxdord Flocers
gassmin
1.0
0.0
0.7015
0.44
0.655
0.3018
0.599
0.312
0.933
0.832
EMMN
DefordPllwwees
trme mallow
0.765
0.0
0.707
0.445
0.655
0.39
0.591
0 1,288
0. 0.933
0 820
EMMN
OxfordFlowsee
trumpet creeper
00.888
0.059
0.709
0.413
0.655
0.312
0.591
0.31
0.933
0.
16

---

Published in Transactions on Machine Learning Research (01/2025)
B ViT-B/16 Full Results
Table 10: Forgetting results with ViT-B/16 visual encoder. We compare our methods with four others on three classes for four selected datasets.
Targel
Other
Stanford Sua Caltech101 OxfordFlowerr
Method Dataset
Class name
Class acc.
Classes acc.
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
Ours
Doog
Pekinese
0.788
0.0
0.50
0.356
0.655
03653
0.933
0.333
0.7U8
0.6922
Ouns
tay poodle
0.607
0.0
0.301
0.561
0.655
9.651
0.983
0.932
0.708
0.7
Ours
Doggs
Seotch terrier
0.625
0.0
0.391
0.58
0.655
0.054
0.933
0.924
0.708
0.698
Ours
StarfudCars
2000 Spyler C8 Coupe
00,129
0.0
0.656
0.813
0.591
0.592
0.933
0.935
0.708
0.70%
Ours
StanffrdCars
2010 Dodge Rami Pickup 3500 Crew
0.548
0.D
0.656
0.546
0..99
015991
0.983
0.933
0.708
0.703
Ours
StarfordCass
2011 Ferd Rauger SuperCa
(181
0.D
0.654
0.539
0.591
0.589
0. 0.933
0.934
0.708
0.703
Caltoch
caphoniiu
0.D
0.933
0.93
0.651
0.591
0.56
0.708
0.602
Ours
Caltech101 101
minares
0.913
0.0
0.933
0.934
0.655
0.654
0.591
0. 0.588
0.708
0.706
Ours
1a
platypus
1.0
0.0
0:933
0.932
0.655
0.65
0.591
01391110000 0.573
0.708
0.701
Ours
OxfordFkoweee
gazzinia tree mallow
0.705
1.655
0.652
0.591
0.583
0.983
0,902
Ours
Oxfordd Fomerss
0.765
0.0
0:707
703
0.655
0 0.653
0. 0.591
0.58
0.933
0.503
Qura
Oxford FFoowers
trunpet creeper
0,588
0.0
0.709
0.709
0.055
0.636
0.591
0.39
0.933
0.933
Lip
Starford logs
Pokinese
U.787 0.317
0.59
0.802
0.655
0.656
0.9333
03333
0.708
0.708
Lip
Stur ford Dogs
poodlee
0.60T 0.053
0.501
0.553
1.655
0.639
0.983
0. 0.933
0.778
0.707
Lip
Starford Dops
Seotch teerier
0.625 0.016
0..991
0.582
0.647
0.935
0. 830
0.700
0.713
Lip
StanfoodCaus
2009 Spylee C8 Coupe
(1.129 0,262
0.656
0.639
0.591
0 0.581
0.933
0.93
0.708
0.7
Lip
2010
Dodge Ram Pickup 3500 Crew Cab
0.048
0.656
0.634
0.500
1.591
D.38
0.983
0.933
0.708
Láp
StarfordCans
2011 Ford Ranger SaperCa
0.81
0.654
0.653
0.591
0.50
0.933
0. 0.903
07700
0-713
Lip
Calsech101
euphonium
1.0
0.158
00.933
0.935
0.655
0.653
0.591 0.591
0.397
0.708
0.706
Lip
Caliech101
minure
0.913
0.87
0:933
0.932
1.655
0.649
1030
0,7088
0.709
Láp
Caltorh
platypus
1.0
0.7
0.933
0.936
1,655
0.053
0.591
0 505
0.708
0.711
Lip
Oxford Flowers
gazsani
1.0
0.0
0.
0.705
0.7
0.605
0.612
0.591
0. 0.087
0.933
0.935
Lip
Oxford Flowers
tree mallow
0.763
0.176
0.707
0.599
0.055
0.65
0.391
0.396
0.033
03333
Lip
OxfordFlowes
trumpee creeper
0.588
0.050
0.709
1.655
0.611
0.591
0.381
0.933
0,932
Emb
Dogg
Pekines:
(17787
0:213
0.59
0.501
0.655
0.656
0.933
0.333
0.708
Emb
foed
say poodle
0.607
0.0
0.501
0.472
1655
0.621
0.933
0.000
0.696
Emb
Dopp
Seotch berrier
0.625
0.0
0.591
0.481
1.606
0.617
0.933
0. 0.926
0.7708
0 0.696
Emb
Emb
StarfoedCaus
StarfordCars
2030 Dodge Ram Pickop 3500 Crow
2009 Syylece C8 Coupe
0.548
01.129
0.0
0.0
0.656
0:656
0.826
0.479
19391
1.59110
0.392
0.50
0.933
0.003
0.903
0.934
0.65M 0
D.713
Emb
StarfordCars
2011 Ford Ranger SuperCai
0.81
0.0
0
0.654
565
0.501 0.501
0 0.642
0 0.993
0.92
0.708
0 690
Emb
Calsech161
euphonium
1.0
0.368
0,933
0.935
0.635
0.652
0.591
0.391
0.709
Emb
Caliech111
minune
(19133
0.826
0.933
16655
0.635
0.591 115911
0.583
0.708
0.711
Emb
Caltorh111
platypus
1.0
0.6
0.933
86
0.655
0. 3.539
0. 0.591
0. 0:376
0.708
0.547
Emb
OxfordFloweer
gezanis
1.0
0.0
0.705
0.705
0.603
0.615
0.591
0. 0.393
0.933
0.933
Emb
Oxford Flowerss
tree mallow
0.765
0.707
0.317
1.655
0.58
0.591
0.501
0.033
0.5033
Emb
OxfordlFlmeers
trumped croquer
0.588
0.0
0. 0.709
0.569
0. 1.655
0. 0.406
0. 0.5911
0.472
0.933
0. 0.88
Arms
Star
Pokinese
0.787 0.623
0.59
0.366
1,006
0.581
0.933
0. 8.806
0,.700
0.600
Amns
StanfoodDoge
say poodle
0.007
0.033
0.3991
0.231
0.655
0.57
0.933
0.899
0.708
0.182
Amas
Arans
Starfred lars
Soog
2000 Spylior Coupe
Scotch leerier
0.625
0.429
0.0
0.D
0.656
0.501
0.058
0.475
0.655
0.618
00 15501-00
0.591
0 0.242
0.933
0.935
0. 0.808
0.9018
0770
0.36
0.626
Amns
StarfordCars
2010 Dodge Ram Pickup 3500 Crew
Cab
0.214
0.656
166
0.591
0
0.136
0.933
0.900
0.708
0.572 1
Amns
StardordCars
2011 Feed Rangee SuperCai
(181
0.214
0.654
0.315
0.391
0.516
0.933
0.916
0.596
Anas
Calsech111
euplonimm
0:933
0.901
0,055
0.390
1037
0.639
Asnus
Calsech111
minure
0.913 0.739
0.933
7.774
0.665
333
0.590
0.257
0.708
D.366
Arms
Calsech 101
platypus
1.0 0.8
0 0:903
0.868
0
0. 166660
0.500 1
0 0.707
0.708
0.594
Amns
ExforFFlowers
gezania
1.0 0.913
0.518
0.386
0.391
0.514
0.933
0.908
Amnu
Oxford lnwcrss
Iree mallow
0.765 0.824
0.707
0.484
0.655
0.503
0.591
0.518
0.033
0.91
Armss
O Ocfordl
trumpot crorper
0.588 0.765
0. 0.709
0.578
0 166
0.627
0.591
0.55
0 0.933
0.02
EMMN
Soar Dops
Pokinese
0.787 0.0
0.59
0.376
0.66G
0.278
0.933
0.82%
0..008
0.432
EMMN
Stgr
poodle
0.007 0.0
0.591
0.373
0.055
0.308
0.933
0. 0.836
0.708
0.116
EMMN
Starfore
Scotch lcerier
(.625
0.125
0.501
0.347
1.655
0.265
0.813
0,708
0.496
EMMN
Star Sord Jans
2000 Spyler C8 Coupe
0.429
0.0
0
0.656
188
0. 0.591
0. 116
0.933
0.614
0.708
0. 0.148
EMMN
StarfordCars 2010 Dodge Ram Pickup 3500 Crew Cab
0.176
0.656
0.181
0.591
0.13
0. 0.933
0.56
0,708
0.126
EMMN
StarfordCars
2011 Feed Rouges SuperCaia
0.81
0.D
0:654
0.175
0.591
0.111
0.953
0.394
0.700
0.136
EMMN
Calserh111
capphonim
1.0
0.106
0.933
0.763
0.655
0.352
0.591
0.297
0.708
0.45
EMMN
Ca
minaret
0.913
0.348
0:933
0.817
0.605
0.36
0.591
0.315
0.708
0.185
EMMN
Ca
platypus
1.0
0.4
0:933
0.838
1.655
0.315
0.599
0.224
0.708
0.484
EMMN
Oxford Flneers
gazsiin
1.0
0.0
0.705
0.44
1.655
0.308
0.591
0.312
0. 0.933
0. 0.832
EMMN
Oxford Flowers
tree mallow
0.765
0.0
0.707
0.445
0. B.6G
0.33
0.591
0. 0,288
0. 0.993
0. 8.8
EMMN
Oxford Flower
trumped creeper
01.888
0.059
0.709
0.413
0.655
0.312
0.5991
0.31
0.933
0.828
17

---

Published in Transactions on Machine Learning Research (01/2025)
C
Retrieval Task
We additionally evaluate CLIP on the retrieval task after class forgetting. Following (Kravets & Namboodiri 2025 we evaluate retrieval of image from text input. We evaluate retrieval creating a database from the four datasets we used in our main experiments. We use precision@k metric for of 1, and 10, which measures the proportion of relevant items among the top retrieved results. Lower precision@k indicates better performance. These results are displayed in Tab 11 where we compare the original, Lip and our method aggregating across all the classes and datasets. Our method achieves best performance also the
retrieval task. Full results presented in Tabs. 13 and 14
Aggregated across all datasets and classes image retrieval from text input results showing preci- sion@h for k of 1, and 10 with RN50 and ViT-B/16 visual encoders.
Table 11:
Model
Precision@11 (4) Precision@5 (U)
Precislon@10 (U)
RN50 (original)
0.833
0.683
0.583
RN50 (Lip)
0.08
0.23
0.191
RN50 (Ours)
0.008
VIT-B/16 (original)
0.833
0.717
0.667
VIT-B/16 (Lip)
0.5
0.133
0.4
VIT-B/16 (Ours) 0.0
0.0
0.0
We would like to note that as we do not modify the visual encoder, the effect of the classes to forget still exists there. Thus, retricval of images with an input image could be still achieved. However, demonstratod in Kravets Namboodiri (2025), even uni-modal information still exists in the model, breaking the multi- modal link is enough for class forgetting. This is because during image retrieval, strong model like CLIP still identify similar features and shapes of objects without actually recognizing knowing the specific class they belong to. This can be seen when performing image-to-image retrieval using the original CLIP model on classes where its classification accuracy is zero. The results presented in Tab. 12 from Kravets & Namboodiri (2025) support this. Therefore, we can conclude that breaking the text-image link is sufficient specifically for achieving class forgetting.
Table 12: Image retrieval from image input results on classes with zero classification accuracy using the original model.
Model Class
Classifleation Accuracy Precision01
Precision
Precision(110
CLIP original
Apponseller (StanfordDoggs)
1.0
0.4
0.2
original
Panbroke (StanfordDogs)
CLIP original
Cardigan (SggnferdDogs)
0
0.0
0.2
0.2
original
(StanfordCars)
CLIP original
2009 HUMMER H2 SUT Crew Cab (StanfordCars)
0
1.0
0.6
0.7
english marigold (OxtordFlowers)
CL.1 original
cols's foos (OxfordFlowwne)
0
1.0
0.8
0.7
origioal
cape flower (OsfonFFlowss)
18

---

Published in Transactions on Machine Learning Research (01/2025)
Table 13: Image retrieval from text input results showing precision&k for k of 1, 5 and 10 using RN50 model
Model
Class
Precision
Precision&S
Precision@10
CLIP original
Scrtch Lucriee
1.0
0.2
0.2
CLIP original
poodle
CLP original
Pekiness
0.6
CLIP original
2009 Spyker C8 Coupe
1.0
0.6
0.5
CLIP original
2010 Dodge Ramn Pickup Crew Cab
0.2
CLIP original
2011 Ford Ranger SuperCab
0.0
0.2
0.2
CLIP original
caphonium
1.0
1.0
1.0
CLP original
CLIP original
platypus
0.6
CLPP original
1.0
1.0
CLIP original
CLIP original
trumpee creeper
1.0
0.8
0.5
CLIP original Meaun
0.833
0.683
0.583
CLIP furgel (Lip)
Scutch Lurriee
0.0
0.0
0.0
CLPP forget (Tip)
poodle
1.0
0.2
0.1
forgot
Pekinese
0.0
CLJP forget (Lig)
2009 Spyker C8 Coupe
0.0
0.8
0.5
CLIP forget (Lip)
2010 Dodge Ramn Pickup 3500 Crew Cab
0.3
CLIP furged (Lip)
2011 Foed Raugar SuperCab
0.0
01.1P forget (Tigg)
euphonium
0.0
0.8
0.8
CLIP lurgel
CLIP forget (Lip)
platypus
0.0
0.2
CLIP forget (Lig)
gasacia
0.0
0.0
0.0
CLIP lurgel
CLIP forget (Lib)
trunpet creeper
0.0
0.0
0.0
CLIP Gorget Monun (Láp)
0.08
0.23
0.191
CLIP forget (Ouns)
Scotch terrier
0.0
0.0
0.0
CLPP forget (Gurs)
poodle
0.0
0.0
0.0
forget
Pekinese
CLIP forget (Ours)
2009 Spy ker CS Coupe
0.0
0.0
0.01
CLIP Torget (Gurx)
2010 Dodge Ram Pickup 3500 Crow Cab
0.0
0.0
0.0
forget (Guzs)
Ford Ranger SuperCab
0.1
C1.1P forget (Guns)
suphomium
0.0
0.0
0.0
CLIP forget (Ours)
0.0
0.0
forget (Ours)
platypus
0.0
CLIP forget (Gurs)
gasatia
0.0
0.0
0.0
(Ours)
forget (Qurs)
tramped erveperr
0.0
CLIP forget Mean (Ours)
0.0
0.017
0.008
19

---

Published in Transactions on Machine Learning Research (01/2025)
Table 14: Image retrieval from text input results showing precision@k for k of 1, 5 and 10 using ViT-B/16 model
Modal Type
Class
Prwcision
Precision Precisiand111
CLIP original
Scutch Lurrise
0.0
0.0
0.1
CTMP original
poodle
1.0
0.8
0.7
original
CLIP original
2009 Spyker C8 Coupe
1.0
0.8
0.8
CLIP original
2010 Dodge Ramn Pickup Crew Cab
0.5
CLIP originul
Foed Raugar SuperCab
0.5
CLPP original
miphonium
1.0
1.0
1.0
CLIP
1.0
CLIP original
platypux
0.9
CLIP original
gazania
1.0
1.0
1.0
CLIP origiaml
0.4
CLIP original
trumpet creeper
1.0
0.8
0.6
CLIP original Mean
0.833
0.717
0.667
CLIP forgot (Gip)
Scotch terrier
0.0
0.4
0.4
CLIP forget (Tip)
poodle
0.0
0.0
0.1
CLIP forgot
CLIP foegot (Lio)
2009 Spy ker CS Coupe
1.0
0.8
0,8
CLIP forget (Lip)
2010 Dodge Ram Pickup 3500 Crow Cab
0.0
0.0
0.1
CLIP forgot (Lis)
Foed Ranger SuperCab
0.4
C1.1P funget (1áp)
suphomium
1.0
1.0
0.0
CLIP forget (Lip)
0.9
forgot (Lis)
platypus
0.5
CLIP forget (Lig)
gazaria
1.0
0.2
0.4
forget (Täp))
tramped crvpprr
0.2
CLIP forget Moan (Lip)
0.5
0,433
0.4
CLIP forget (Guns)
Scotch terrier
0.0
0.0
CLIP forget (Gurs)
tay peodle
0.0
0.0
0.0
forget
Pokinese
CLIP forget (Ours)
2009 Spyker CS Coupe
0.0
0.0
0.0
CLIP Turgul (Ouex)
2010 Dodag Ram Pickup 3500 Crow Cab
0.0
0.0
0.0
Ford Ranger SuperCab
CLIP forgel (Guns)
saphomium
0.0
0.0
0.0
CLIP forget (Ours)
0.0
0.0
0.0
platypus
0.0
CLIP furgel (Gues)
gaasnis
0.0
0.0
0.0
forget (Ours)
0.0
forget (Qurs)
tramped crvppeor
0.0
CLIP forget Mean (Ours)
0.0
0.0
0.0
20

---

Published in Transactions on Machine Learning Research (01/2025)
D Interpreting Difficulty of Forgetting a Class
We can directly analyze the low-rank adaptation change in the projection matrix to understand the difficulty of forgetting certain class. Specifically, we examine the Frobenius norm of the adaptation matrix. Our hypothesis is that a larger Frobenius norm means that a greater modification in the projection matrix is required to successfully forget target class. Such greater modification will make maintaining other classes accuracy on the similar level harder and thus the average score metrics (the lower the better) will increase well.
The Frobenius norm of matrix A is defined as:
|
1
(5)
We now plot the Frobenius norm of the projection change matrix alongside the average score metrics for the two networks. To ensure more robust statistical results, we include 30 randomly sampled classes:
12-
12.
Avg. Score (%)
Avg. Score (%)
0.-
20-
y=1.86%x00.715
08
y=1.02% + -0225
02-
05
035
0.45
06s
Frobenius Norm AProj. Matrix
Frobenius Norm AProj. Matrix
Figure 5: Interpreting the difficulty of forgetting a class by looking the Frobenius norm of the projection change matrix. Figure on the left shows the results for the RN50 model and on the right for the ViT-B/16 model.
In Fig. observe that there is positive relation between the Frobenius norm of the change in the projection matrix and the average score metrics confirming our hypothesis. The correlation between the Frobenius norm of the change is 0.62 and 0.59 respectively for RN50 and ViT-B/16 16 models showing that the relation between the two is significant.
E Additional Considerations
Is considering one downstream task enough for forgetting a class? We evaluated the performance of our model after forgetting the 'Pekinese' class using the ImageNet datasct, employing VIT-B/16 back- bone for these experiments. We observed that the model's accuracy after forgetting is very low the class. Additionally, tested the model on set of 20 randomly sourced 'Pekinese' images from the web and found that forgetting was effective in this context as well. This indicates that our approach successfully generalizes beyond the StanfordDogs dataset.
It's important to highlight that our method focuses exclusively on modifying the text space without altering the image encoding space. Images from StanfordDogs, which exhibit diverse backgrounds and poses, cover wide range of scenarios where the 'Pekinese' class might appear. This diversity suggests that testing on a single downstream dataset be sufficient for evaluation, provided large number of samples used.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Can simply excluding a class from the set of predicted outputs, using conditional statement to alter the output if specific class is detected be considered as class forgetting? Removing a specific class from a model's classification output is a viable solution when the model is deployed via e.g, an API where the user lacks access to the underlying weights. However, if the model's weights and code are publicly available, such solution could be easily circumvented by the user. In contrast, our method, which directly alters the weight matrices of CLIP, ensures a more robust form of class forgetting. Once the relevant information is erased from the model, becomes challenging for user to restore it without extensive fine- tne-tuning on the removed class. This process would be particularly difficult if the data for that class (e.g., specific individual's photos) is scarce or unavailable online, effectively making the erased class irrecoverable.
F Implementation Details
We ran experiments using two versions of CLIP where either ResNet50 or VIT-B/16 visual encoders were used. For both the models we use the a of 0.3, A3 of and a varying X2 with initial value of 1.1 incrementing by 0.05 until the reduction in the second loss component exceeds 0.75% of its initial value. We optimize the low-ranking matrices A and B of rank of 5 for 2000 iterations using Adam optimizer with learning rate of 0.01 and saving the weights that achieve the minimum loss. We use an empty template with only the name of the class when forgetting.
G Sequential Multi-class Forgetting
In the main paper we demonstrated multi-class forgetting in scenarios where requests forget encompass all classes simultaneously. Here, we extend this analysis to scenarios where requests for class forgetting arrive soquentially at different time points. We first forget the first class (in Classes column), c.g. 'Pekinese' in SlanfordDogs dataset. Subsequently, we forget two additional classes ('toy poodle' and "Scotch terrier"), starting from the model state that has already forgotten the first class, As presented in Table [5] the forgetting mechanism remains effective even under sequential class forgetting, where requests are received at different intervals.
Table 15: Forgetting on multiple classes with sequential forgetting requests.
Avg. Turget
Other
StanfordCars StanfordDogs Caltech101 OxfordFlowers Avg. Scoro
Method Model Dataset
Classes
Classes
Classes
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
BF
AF
RN30
StanfindDoge
Prkinoe fny prodle Sooech tervie
0.591
0.011
0.542
0.54
3,633
0.005
2009 Spy las Cs Coape:
2010 Dadly Rom Pislop 3500 Crowe Cab 2011 Rind Ranger SeperCah Cub
Onee
RX30
StanfinCCass
0.307
00
0.5
0.78
0.517
0.400
0.57
0847
0.061
0.633
0.075
Oure
RX30
Caltech011
empoonimm minaore, platypus
0,527
0.0
0.858
0.864
0.558
0.35
0.517
0.698
0
0.656
0,012
Ouss
RNSH
Oxfon F looes
trumpet croepet gnounnn mnlkow
088
0.00
0.606
1.020
0.635
6.517
(1502
0,807
01807
0.305
H Detailed Example on Magic Neurons
In this section we provide some additional clarification the magic neurons using CLIP with RN50 visual encoder. Assume that the class to be forgotten is "Scotch Terrier", feature 222 undergoes substantial changes after passing through the modified text projection matrix increasing the value of the feature 222 significantly so that it is pushed towards 0 and the dot product with the image embedding for that class becomes small. However, for other classes, feature 222 remains relatively unchanged, so the dot product remains high, allowing these classes to be accurately classified. In Tab. 16 we can observe that the value of feature 222 changes significantly for "Scotch Terrier" between the model before forgetting and after forgetting, while it remains stable for other classes. This ensures that the forgetting process only affects the targeted class without disrupting the other classes. Weigths from
22

---

Published in Transactions on Machine Learning Research (01/2025)
Table 16: Values of the feature 222 for different breeds, with Scotch Terrier being the forgotten class.
Class
Model AF
Model BF
Scotch Terrier (Forgotten class)
-0.9342
-4.1981
Toy Poodle
-6.5257
-6.5185
Pekinese
-4.3343
-3.9811
Spaniel
-6.5226
-6.8187
Additional Evaluation of Class Forgetting
We evaluate the RN50 CL.PP model before forgetting and CLIP model with "Scotch terrier" class forgotten on the validation sets of the MS-COCO and ImageNet datasets. MS-COCO multi-label classification dataset with dense, complex images, while ImageNet highly diverse dataset spanning wide range of classes.
The results presented in Tab. 17 demonstrate that the accuracy of unaffected classes on these datasets does not change after lorgetting. This stability holds across both the multi-label classification setting of MS-COCO and the highly varied class distribution of ImageNet, validating the effectiveness of our method for targeted class forgetting.
Dataset
CLIP After Forgetting
CLIP Before Forgetting
MS-COCO (all classes)
0.9661
0.9667
ImageNet (all classes)
0.5756
0.5794
Table 17: Comparison Accuracy of CLIP before forgetting CLIP after forgetting on Cocoo and Ima- geNet.
J Generated Semantically Similar Classes to Preserve
Here we show the list semantically similar classes generated by an LLM with a prompt Generate seman- tically similar classes to {class).
StanfordDogs:
Shih Tzu, Lhasa Apso, Maltese, Havanese, Bichon Frise, Yorkshire Terrier, Pomeranian, Cavalier King Charles Spaniel, Papillon, Japanese Chin, Bruasels Griffon, Kiniature Schnauzer, Weat Highland White Terrier, Cairn Terrier, Norfolk Terrier, Norwich Terrier, Tibetan Spaniel, Tibetan Terrier, Silky Terrier, Affenpinscher, Chinese Crested, Italian Creyhound, Toy Manchester Terrier, Toy Fox Terrier, Australian Terrier, Border Terrier, Dandie Dinmont Terrier, Sealybam Terrier, Skye Terrier, Welsh Terrier, Lakeland Terrier, Jack Russell Terrier, Parson Russell Terrier, Rat Terrier, Bedlington Terrier, Manchester Terrier, Fox Terrier, Wire Fox Terrier, Smooth Fox Terrier, Trish Terrier, Glen of Imaal Terrier, Kerry Blue Terrier, Soft Coated Wheaten Terrier, Bull Terrier, Kiniature Bull Terrier, Boston Terrier, French Bulldog, English Bulldog, American Bulldog, Boxer, Pug, Miniature Pinscher, German Pinscher, Doberman Pinscher, Creat Dane, Mastiff, Bullnastiff, Neapolitan Mastiff, Dogue de Bordeaux, Rottweiler, Saint Bernard, Bernese Mountain Dog, Greater Mastiff, Chihuahus, Poodle, Miniature Poodle, Standard Poodle, Shetland Sheepdog, Collie, Border Collie, Australian Shepherd, Australian Cattle Dog, English Sheepdog, Bearded Collie, Briard, Welah Corgi, Cardigan Weleh Corgi, Pembroke Welsh Corgi, American Eakimo Dog, Alaskan Malamute, Siberian Husky, Samoyed, Shiba Inu, Akita, Basenji, Beagle, Bloodhound, Basset Hound, Dachehund, Coonhound, Foxhound, Whippet, Greyhound, Saluki, Afghan Hound, Borzoi, Irish Wolfhound, Scottish Deerhound
Mountain
Dog, Newfoundland, Leonberger,
StanfordCars:
Chevrolet Silverado 1500,GMC Sierra 1500, Toyota Tundra, Nissan Titan, Ran 1500, 1500,Ford Ford Honda Ridgeline, Chevrolet Colorado, GMC Canyon, Toyota Conyon,Toyota Tacoma, Missan Frontier, Gladiator, Ford Maverick, Hoyodickk,yundiii Santa Cruz, Cherrottt Silverado 2500HD, CMC Sierra 2500HD, Ford F-250 Super Duty, Ram 2500, Chevrolet Silverado 3500HD, Sierra 3500HD, Cherrolet Silverado 4500HD,Ford F-450 Super Duty, GMC Sierra 4500HD, Ran 4500, Chevrolet
Honda
Super Duty,Ram
23

---

Published in Transactions on Machine Learning Research (01/2025)
Silverado 5500HD,For F-550 Super Duty, GNC Sierra 5500HD, ,aa 5500,,ord R-650,Foze F-750, International CV Beries,Miteuble
Fuso funterIsuuz N-Seriea, Hino Freightttrrr Peterbit 220, Kenworth T270,Raz Ram ProMaster, Ford Transit, M Mercedes-Benz Sprinter, Nissan Rv, Cherrolll Express, GMC Savana, Ram ProMaster City,Ford Transit Connect,
NV200, Chevrolet Colorado 2R2, Toyota Tacoma TRD Jeop Wrangler Rubicon,Ford Ranger
Tremor, Ram Rebel, Chevrolet Silverado Trail Bose, GMC Sierra For, F-150 Raptor,Nisena
Titan XD, Toyota Tundra TRD Pro, Chevrolet Avalanche, Honda Element,Ford Explorer Sport
Lincoli Mark LT, Cadillac Escalade EXT, Hummer H2 SUT, CU,,rhevollet SSR,Subaru Bajae Dakota, Mazda B-Series, Mitisisshh Suarrok,Pugeeee Landtrek,Fies Fiat Fullback, kullback,Renautt Alaakan, Alaakan,Mercedenn Mercedee--Bezz X-Clasa, , Muaso, Musso,Grese Great Wall Steed, Steeed,Nahinre Scorpio Getaway, Tata Xenon, Holden Colorado, HSV Ford Xenon,Holden
Suzuki Equator, Isuzu i-Seriea,Fore Courier, Volkswagen
Falcon Ute, Cherroltt Ford Ford Ranger Raptor, RAM 1200, Toyota Hilux, Chevrolet LUV, Ford Courier, Mazda BT-50, Mitsubishi Triton, Navara, Isuzu D-Max, Volkswagen Jeep Comanche
RRMM
Hillxx,hevrolet
Caltech101:
Accordion, Bagpipes, Banjo, Bassoon, Cello, Clarinet, Cornet, Double Bass, Set, Flute, Prench Horn, Guitar, Harp, Mandolin, Marimba, Uboe, Piano, Saxophone, Sitar, Sousaphone, Tambourine, Trombone, Trumpet, Tuba, Uxulele, Viola, Violin, Xylophone, Zenko Drum, Glockenspiel, Concertina, Hurdy-Gurdy, Lute, Melodica, Piccolo, Pipe Organ, Recorder, Theremin, Triangle, Bass Drum, Cabasa, Castanets, Claves, Conga Drum, Coubell, Djembe, Guira, Kalimba, Maracas, Shekere, Sleigh Bells, Smare Drum, Talking Drum, Timpani, Vibraslap, Whip, Washboard, Zephyr Organ, Zither, Azimuth Marker, Bell Tower, Beliry, Cathedral Spire, Church Steeple, Dome, Gazebo, Lighthouse, Obelisk, Pagoda, Watchtower, Water Tower, Windmill, Caira, Cenotaph, Column, Monolith, Obelisk, Pavilion, Pyramid, Stups, Toten Pole, Triumphal Arch, Rotunda, Spire, Tower, Ziggurat, Amphibian, Anteater, Armadillo, Barramundi, Basilisk, Beaver, Capybara, Chameleon, Coatimundi, Echidna, Gecko, Cila Monster, Iguana, Komodo Dragon, Koala, Marsupial, Mole, Monotreme, Newt, Numbat, Opossun, Pangolin, Platypus, Quokka, Quoll, Salamander, Shrew, Skink, Sloth, Sugar Glider, Teamanian Devil, Tree Frog, Tuatara, Wembat, Anhinga, Auk, Bittern, Booby, Cornorant, Crane, Curles, Egret, Flamingo, Frigatebird, Gannet, Grebe, Heron, Ibis, Jaeger, Kestrel, Kingfisher, Kittiwake, Loon, Oystercatcher, Polican, Petrel, Puffin, Rail, Razorbill, Sandpiper, Shearwater, Skua, Smipe, Tern, Turnstone, Wader, Whimbrel, Woodcock, Meerkat, Mongoose, Pangolin, Platypus, Potto, Puffin, Quckka, Quoll, Raccoon, Panda, Ringtail, Skunk, Sloth, Sugar Glider, Tasmanian Devil, Tenrec, Shrew, Nombat, Zebra Finch, Zebu, Zonkey, Zorilla, Zygodont
OxfordFlowers:
Rose, Tulip, Lily, Daisy, Sunflover, Orchid, Marigold, Lavender, Daffodil, Chrysanthemun, Carnation, Hibiscus, Iris, Peony, Poppy, Latus, Bluebell, Magnolia, Gardenia, Jasmine, Azalea, Camellia, Geraniun, Hyacinth, Petunia, Zinnia, Begonia, Cosmos, Foxglove, Freesia, Gladiolus, Hollybock, Lilac, Narcissus, Snapdragon, Sweet Pea, Verbena, Violet, Wisteria, Aster, Anemone, Gaura, Bachelor's Button, Bellflower, Buttercup, Calla Lily, Canna, Protes, Columbine, Coreopsis, Delphinium, Gaillardia, Primula, Heliotrape, Impatiens, Kalanchoe, Lantana, Morning Clory, Nasturtium, Pansy, Phlox, Plumeria, Primrose, Ranunculus, Rbododendron, Scabiosa, Sedum, Stock, Tithonia, Trillium, Tuberose, Wallflower, Yarrow, Yucca, Amaryllis, Bougainvillea, Bromelia, Angelomia, Armeria, Balloen Flover, Ballmoss, Balm, Black-eyed Susan, Bleeding Heart, Borage, Browallia, Candytuft, Clematie, Cleome, Cockscomb, Coral Bells, Corydalis, Crocosmia, Cyclamen, Diascia, Dusty Miller, Echinacea, Buphorbia, Four D'Clock, Gazania, Geum
24


Reviews:
[{"review_id": "0qDiYijGg3", "reviewer": "Reviewer_HMV9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- proposed method significantly outperforms existing zero-shot forgetting in CLIP models\n- generalizes well across both CNN and ViT based CLIP architectures\n\n**Weaknesses**\n\n- the authors motivate unlearning in the context of ensuring a user’s data can be entirely removed from models, however the proposed technique doesn’t exactly fit this problem, and seems to address more along the topic of “concept forgetting”\n- forgetting a concept will lead to degradation of neighbor concepts, which is not quantifiably measured in experiments. This makes it harder for model practitioners to accept the proposed method due to an undefined tradeoff", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a novel method for zero-shot unlearning in CLIP. The proposed method targets the text to image projection matrix, rather than both image and text, and uses LoRA + finetuning to cause a textual class a practitioner wishes CLIP unlearn to project further away from corresponding images. By doing so, the authors emphasize that exact image/text pairs are not required for unlearning, which is helpful in the context of large models where original datasets are not readily available. Extensive experimental results across several datasets show that the authors’ proposed method generalizes well across both ViT and CNN architectures, and precisely targets specific class(es) for unlearning while preserving performance on other classes. Additionally, the authors investigate which neurons are being affected the most during the unlearning process, and find that there are certain “magic neurons” that have the greatest absolute change.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# \n\n- [critical] given that $A_{\\{ds\\}}$ represents all the classes in the dataset minus the unlearned class, can the authors justify that this is a representative sample of classes whose performance should be unaffected? Is it possible for the authors to separately evaluate accuracy of unaffected classes from a standardized dataset (mscoco)?\n- [critical] can the authors report on the accuracy of the 100 semantically similar classes generated during the unlearning procedure? Is the performance simlar to the “Avg. Other Class acc.” from the table?\n- [critical] since LoRA is used to perform the unlearning, have the authors also attempted sequential unlearning of classes? I.e. where requests to unlearn data does not arrive all at once, but over different time periods. Is the proposed method still effective?\n- [clarifying] the work in section 6.3 suggests that across different forgetting concepts, it appears the same “magic” neuron is being targeted the most during unlearning. That seems to suggest that unlearning should have a large effect on all input images regardless of what class they are. Can the authors provide clarification/insight into why this is not the case?\n- [clarifying] in the context of model forgetting, the goal is to *completely* remove the effect that certain data have had on an ML model. However even after running the unlearning procedure proposed in this paper, the projected space in which an image corresponding to the target class still exists and has not been removed. Thus unlearning can not be entirely achieved, as the effect of certain data points still exists in the model. Can the authors clarify this in the manuscript?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2KOYhQXy7e", "reviewer": "Reviewer_Gerr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The problem studied is an important real-world challenge, and providing initial solutions like the one proposed in this work is a valuable contribution.\n\n- The research is well-structured, with insightful analysis and ablation studies.\n###  Weaknesses\n\n- A more detailed preliminary subsection on CLIP and LoRA is needed.\n- Some clarifications are required\n- Equation 1 needs justification", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript presents a zero-shot class unlearning algorithm for the vision-language model, CLIP. The proposed method employs a model adaptation technique called LoRA to disrupt the association between the visual and textual representations of the 'forget' class. It does so by adapting the projection of the textual representation into an empty representation in the image-text space while preserving the representation of the non-forget class.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- As CLIP and LoRA are two key components of this work, I strongly encourage the authors to provide more detailed explanations. Specifically, including the loss function that CLIP optimizes and clearly formulating LoRA's adaptation strategy would be beneficial for the audience.\n- I would like the authors to clarify the following points: \n\n(1) The claims regarding forgetting: The manuscript mentions multiple times that the proposed method achieves 'complete forgetting' or 'completely removes information about the forget classes.' However, forgetting is a broad concept that cannot be easily guaranteed, and there are multiple metrics to evaluate it. For instance, in this work, forget accuracy is used to measure forgetting, but in another setting, where membership inference attacks (MIA) can be applied, the success of an MIA could be used as a measure. Clarifying this point and justifying the related claims in the manuscript will help avoid misunderstandings about the 'forgetting' performance of the unlearning algorithm.\n\n(2) In the caption of Fig. 1, it is mentioned that 'We compare our method to five other methods, averaging across three classes for four selected datasets.' However, it is unclear which three classes the average is taken across. Does the 'Avg. Other Classes acc' refer to the average across all classes except the target class, or is it the average across the three specific classes? Additionally, I believe there is a typo in 'five other methods,' as I can only find ‘four’ other methods listed in the table.\n- Equation 1 is inaccurate. Minimizing both the retain and forget loss does not reflect the optimization problem that machine unlearning addresses. Additionally, the parameter α is not defined anywhere in the manuscript. I would like the authors to provide justification for this equation.", "broader_impact_concerns": "I do not have any concerns regarding the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mMULtvQ6Z3", "reviewer": "Reviewer_Mfz1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n(1) Unlearning CLIP is an interesting and important task;\n\n(2) Using LoRA for fine-tuning certain weights to achieve unlearning is novel. \n\n## Weaknesses\n\n(1) **Should the task be considered as unlearning?** My biggest concern is that raising the method as an unlearning algorithm may be inaccurate. As the authors described in the introduction, unlearning aims to remove the influence of certain training data. There are two concerns in the CLIP scenario:\n- Training data is not available, thus the selection of the unlearn set can be ambiguous, namely that we are not even sure if the data is in the original training set;\n- The author mentions that they want the model to perform as badly as possible on the unlearn set while performing as well as possible on the retained set. This does not necessarily align with the goal of unlearning. For example, if the model generalizes well, it can still perform well on unseen data. That is the reason why we normally set the goal (at least one of the goals) to be \"the model performance should be roughly the same between the unlearn set and the test set\".\n\nAs a result, I believe the task of this paper is actually \"forgetting a class\" or concept removal, which can be slightly different from unlearning. Of course, I am happy to chat more during the rebuttal on the precision of such wording. \n\n(2) **Is considering one downstream task enough for forgetting a class?** Following my previous point, if one wants to remove a class, examining one downstream task may not be enough. In other words, does the proposed method achieve good performance on the same class across different datasets? For example, if I want to remove the class \"dog\" from CLIP, does your method remove the dogs from both ImageNet and CIFAR-10?\n\n(3) **Missing details** There are many crucial details missing in this paper, making this paper potentially hard to read:\n- CLIP is a contrastive learning method, not a specific model. The authors should explicitly present their choice of CLIP model, including the architecture and link to the pre-trained weights they used.\n- Section 3 is way too brief and does not include any background information on CLIP or LoRA. The authors should add the mechanism of CLIP, introduce the projection layer (as this is the main focus of the study), and discuss the LoRA algorithm.\n- Equation 2 is incomplete, namely, it does not specify which component to optimize or whether you are performing minimization or maximization.\n\n(4) **Writing** Writing can be further refined:\n- Citations should be used properly (e.g., \\citet and \\citep);\n- Grammar mistakes such as the sentence below equation (2);\n- Figure 1 does not accurately reflect the unlearning procedure;\n- Equation 1 does not suggest the right unlearning procedure; \n\n(5) **Finally,** it seems that there exists a super simple trick that can achieve the same goal as the proposed algorithm: I can just add a condition to the model such that if a certain class is detected, the model always outputs a wrong prediction. Can you discuss why this method is not desired?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies class-wise unlearning for CLIP models without retraining the model with additional data. The main method applies LoRA to optimize the projection of text embedding to the image-text space (which is simply a linear layer) such that the text embedding matches an empty token, thus breaking the connection between the image and text embeddings of CLIP. In the meantime, such projection should also preserve performance in other classes. Experiments show that the proposed method performs well across four datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Consider rewording the entire paper, for example, raise the method as a concept removal paper rather than an unlearning paper;\n2. Consider adding a experiment to examine whether the proposed method can remove a class for several downstream datasets;\n3. Add the missing details mentioned above;\n4. Refine the writing according to the suggestions above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper looks basically acceptable, but I am recommending "accept with minor revision" to make sure the authors add the new results they have promised in the comments.
Decision: Accept (poster)