MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation
Authors: Yihan Wang
Yiwei Lu
Guojun Zhang
Franziska Boenisch
Adam Dziedzic
Yaoliang Yu
Xiao-Shan Gao
Venue: Accepted by TMLR ()

Abstract: Machine unlearning offers effective solutions for revoking the influence of specific training data on pre-trained model parameters. While existing approaches address unlearning for classification and generative models, they overlook an important category of machine learning models: contrastive learning (CL) methods. This paper addresses this gap by introducing the Machine Unlearning for Contrastive Learning (MUC) framework and adapting existing methods. We identify limitations in current approaches, noting that several methods perform inadequately as unlearners and that existing evaluation tools insufficiently validate unlearning effects in contrastive learning. To address these issues, we propose Alignment Calibration (AC), a novel method that explicitly considers contrastive learning properties and optimizes towards new auditing metrics for easy verification of unlearning. Through empirical comparisons with baseline methods on SimCLR, MoCo, and CLIP, we demonstrate that AC: (1) achieves state-of-the-art performance, approximating exact unlearning (retraining); (2) enables data owners to clearly visualize unlearning effects through black-box evaluation. The code is available at  https://github.com/EhanW/Alignment-Calibration.

Published in Transactions on Machine Learning Research (08/2025)
MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation
Yihan Wang University of Waterloo
ythan.wang wang@nuuaerroo.cc
Yiwci Lu University of Ottawa
yéwei. Qunneiiknuuotttuuo.oo
Guojun Zhang Alibabo
Franziska Boenisch
boenisch@cispa.cc de
CISPA Heimholtz Center for Information Security
Adam Dziedzic
dziedzic de
CISPA Helmholtz Center for Information Security
Yaoliang Yu University of Waterloo Vector Institute
gaoliang, prolianngguuuwwaterii ww@uwaterloo.cc ca
Xiao-Shan Gao
egao@@mmrc.iss.cccoom
Academy of Mathematics and Systems Science, Chinese Academy of Sciences University of Chinese Academy Sciences
Reviewed OpenReview: https: openreview. net/foruee id= j9D
Abstract
Machine unlearning offers effective solutions for revoking the influence of specific training data on pre-trained model parameters. While existing approaches address unlearning for classification and generative models, they overlook an important category of machine learning models: contrastive learning (CL) methods. This paper addresses this gap by introducing the Machine Unlearning for Contrastive Learning (MUC) framework and adapting existing methods. We identify limitations in current. approaches, noting that several methods perform inadequately unlearners and that existing evaluation tools insufficiently validate unlearning effects in contrastive learning. To address these issues, propose Alignment Calibration (AC), novel method that explicitly considers contrastive learning properties and optimizes towards new auditing metrics for easy verification of unlearning. Through empirical comparisons with baseline methods on SimCLR, MoCo, and CLIP, we demonstrate that AC: (1) achieves state-of- of-the-art performance, approximating exact unlearning (retraining): (2) enables data owners clearly visualize unlearning effects through black- box evaluation. The code is available at https: https:///gtthuue wwt.ggt.......aaaieeeeeeeeeeeeae http//aaat///ggggennaaaaaabbbeea
1
Introduction
The success of modern machine learning models largely relics on training with a large corpus of data. However, carefully annotated data are expensive and difficult to obtain, thus urging the utilization of the vast amount.
"Equal contribution
Work done while Huawei.

---

Published in Transactions on Machine Learning Research (08/2025)
of unlabeled data in the wild. The recent self-supervised learning methods, especially contrastive learning methods (Chen et al., 2020; 2021; He et al., 2020), provide viable solutions to learning general representation for various downstream tasks. For example, unimodal contrastive learning models employ the InfoNCE loss to maximize the feature similarity between positive pairs (e.g., different data augmentations of the same image) while minimizing that between the negative ones (e.g. different images). This training scheme also applics to multi-modal training (e.g., CLIP (Radford et al., 2021)); and the learned encoders are widely applied in various tasks, e.g., GPT-based models (Achiam et al., 2023) and latent diffusion models (Rombach et al., 2022).
To amass large-scale datasets for training contrastive learning models, practitioners often resort to web crawling (e.g., using Common Crawl*). However, such data collection methods may disregard data owners' privacy concerns, potentially retrieving their data without consent. Moreover, acquired training data may include copyrighted material or even inappropriate content, such as sexual abuse (e.g., in recent reports? against content in LAION-5B (Schuhmann et al., 2022)). In these scenarios, data owners or authoritics may right/ully request the removal of misused training data (i.e., unlearning dataset) 3, necessitating adjustments to the trained model parameters. While retraining the model from scratch without the unlearning dataset is straightforward solution, it incurs substantial computational costs for large models and datascts.
To eliminate the effect of the unlearning dataset on the model with minimum effort, machine unlearning methods (Cao & Yang, 2015; Bourtoule et al., 2021; Ginart et al., 2019; Guo et al., 2019; Neel et al., 2021; Ullah et al., 2021; Sekhari et al., 2021; Izzo et al., 2021; Chen et al., 2023; Zhang et al., 2024a; Fan et al., 2024; Shen et al., 2024) provide recipes for supervised learning methods on group removal and for generative models sample or concept removal. However, the study of an efficient solution for contrastive learning models is under-explored'. In this paper, establish the foundation of Machine Unlearning for Contrastive Learning models (MUC). MUC adapts various existing methods to contrastive learning and introduces the notion of data owners who request unlearning and model owners who execute unlearning. Given candidate unlearning algorithms, the model owners first perform white-bor evaluation to select the best method and generale an optimal unlearned model. The data owners then perform black-bou evaluation to validate the effect of the unlearning procedure. We argue that unlearning success is achieved only if the unlearned model meets the criteria on both sides. We summarize this unlcarning pipeline in Figure 1.
Unfortunately, direct adaptations of existing unlearning approaches are unsatisfactory on both considerations. Firstly, from the model owners' perspective, such algorithms are suboptimal approximations of exact unlearning (training from scratch) under different white-box evaluations and there lack of a good candidate method. Secondly, from the data owner's perspective, even given an optimal unlearned model, it is difficult to discern the unlearning effect under existing black-bor evaluation tools, rendering hard to determine the success of unlearning.
Motivated by the above state of affairs, we introduce novel unlearning method called Alignment Calibration (AC) that is specifically tailored for contrastive learning. AC optimizes novel loss function that involves three terms: (1) positive aligmment calibration term that removes the footprint of the unlearn set on the representation; (2) a negative aligmment calibration term that leaves explici unlearning traces for evaluation; (3) a performance preserving term that maintains uniformity.
Finally, we empirically compare bascline methods with our Aligument Calibration algorithms on unlcarning models pre-trained on SimCLR (Chen et al., 2020) MoCo (He et al., 2020), and CLIP (Radford et al., 2021). ). Under various unlearning settings (e.g., the fraction of the unlearning dataset) and evaluation metrics, AC consistently outperforms the bascline methods, especially under unlearn evaluation, validating the benefits of our method. In summary, we make the following contributions:
'attpe://commooceaall org/
Phttps://purl http:////prrsssatord...ooo 2du//kh75em9123
IIn accordance with policies such the European Union's is General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Canada's proposed Consumer Privacy Protection Act (CPPA).
WWe note that Wang et al. (2023) addresses related problem in contrastive unlearning. However, it differs siguifieantly from our approach in several key aspects. Notably, Wang et al. (2023) does not provide comprehensive evaluation framework or establish strong baseline comparisons.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Unlearning Request
Model Owner
Data Owners
Unlearning Evaluation
Unlearned Model
Input:Pretrained Encoder
Onteut:Unlearned Encoder
Before Unlearning
After Unlearning
Unlearning Algorithm
Unlearning Auditing
g
g
( unlearn
g
unlearn/
How to choose optimal algorithm?
Evaluate the effect of unlearning
D unlearn 9:
Access:
D Prrainy D unlearr; Dteat: 9
Access:
Principle:
Approximate exact unlearning with efficiency
Principle:
Comparing the model output on the unlearn subset
Unlearning traces through
Method:
Alignment Calibration (our method)
Solution:
Alignment Calibration (our method)
Figure 1: An overview of the unlearning pipeline for contrastive learning. Given training set Perain contains an unlearn subset Dunlearn, and pretrained encoder 9. model owners (Column 1) aim to find an optimal unlearning algorithm by comparing with exact unlearning (retraining with the retain dataset Dtrain / (mnlearn) to obtain an unlearned encoder 9. Data owners (Column 2), who only have black- bor access to the model outputs, then evaluate the unlcarning outcome by comparing the output features bcforo/ /after unlearning.
We propose the MUC framework that considers existing methods and various evaluation tools in contrastive learning, including while-box evaluation and black-bor evaluation.
Motivated by insufficiencies of existing unlearning algorithms and evaluation tools, we propose the novel Alignment Calibration method that satisfies both model owners and data owners.
Our experiments initiate the evaluation of existing machine unlearning methods for contrastive learning and confirm the superiority of our new methods.
2 Background and Related Work
We first provide background and related work on contrastive learning and machine unlearning.
Contrastive Learning and Self-supervised Learning Contrastive learning learns general representations
by contrasting sample pairs, which analytically benefits the downstream applications (Saunshi et al., 2019; Tosh et. al., 2021). Popular contrastive learning methods such as Contrastive Predictive Coding (CPC) (van den Oord et al., 2018), SimCLR (Chen et al., 2020), and MoCo (He et al., 2020) employ the InfoNCE loss to enforce the contrast between positive and negative pairs. Other variants of the InfoNCE-based loss are also widely applied, e.g., f-MICL (Lu al., 2023), Alignment and Uniformity (Wang & Isola, 2020), and Pearson x² divergence (Tsai et al., 2021). This contrastive training scheme also applied to the context of multimodal learning, where images and texts are formed pairs, e.g., in CLIP (Radford et al., 2021). There exist other self-supervised learning methods that also learn representations (Grill et al., 2020; Chen & He, 2021; He et al., 2022; Caron et al., 2021). In this paper, we mainly focus on developing unlearning recipes for contrastive lcarning methods, especially SimCLR, MoCo, and CL.P.
Specifically, contrastive learning usually applies the InfoNCE loss to learn a representation g. Given a probability measure p, we define the density of positive pairs sampled from p as p+, i.e., two samples with similar feature embeddings as the joint distribution. Specifically, minimizes the loss below the objective:
LIatOKCE =
E
s(g(z).g(v)) + E log E exp((s(g(z), xap yap
(1)
(a.))ppt
3

---

Published in Transactions on Machine Learning Research (08/2025)
where s is the cosine similarity after normalization with temperature parameter, and g(z). are the features extracted by given encoder y,  respectively. The above contrastive learning (pre-training) scheme learns a general encoder 9 (image and text encoders for CLIP). Such a (fixed) can be utilized with an additional linear head or shallow models for downstream tasks. In this paper, we mainly consider linear probing, where g is used for the classification of the same dataset with pretraining. Notably, we consider unlearning during the pretraining phase only.
Machine Unlearning For Supervised Learning: Machine unlearning (MU) (Cao & Yang, 2015) requires algorithm to revert to state that specific data points are never trained on. While exact unlearning (Bourtoule et al., 2021) (e.g,, retraining the model entirely the retain dataset) provides reliable solution, the additional computation requirement is also tremendous. In this paper, we focus on approximate unlearning (Ginart et al., 2019; Guo et al., 2019; Neel et al., 2021; Ullah et al., 2021; Sekhari et al., 2021; Izzo et al., 2021; Chen et al., 2023; Zhang ct al., 2024a; Fan ct al., 2024; Shen et al., 2024) to efficiently achieve the same goal.
For Generative Models: MU methods are applied to diffusion models to avoid copyright infringement and inappropriate image generation (Gandikota et al., 2023; Zhang et al., 2023b; Heng & Soh, 2023; Kumari et al., 2023). For large language models, MU is applied as model-editing (Yao et al., 2023) tool to enable forgetting on certain training texts (Mitchell et al., 2022b;a; Jang et al., 2022; Eldan & Russinovich, 2023; Zhang et al., 2023a; Hu et al., 2024; Jia et al., 2024; Maini et al., 2024; Liu et al., 2024; Zhang et al., 2024b). In this paper, we focus on MU on supe-super learning, specifically, contrastive learning methods, which differs from the above two cases both unlearning settings and frameworks, which we specify in the following section.
3 Machine Unlearning for Contrastive Learning (MUC)
In this section, we specify the problem setting of machine unlearuing for contrastive learning, introduce direct. adaptations of existing methods, and propose evaluation metrics.
3.1 Problem Settings
Formal Notations: (1) We denote the pretrained encoder as g and the encoder after unlearning as g- Given an input sample x, the features extracted by the encoders are denoted as g(x) and ğ(x) respectively; (2) We denote the original training set as Dersin (which is used to train g), the unlearn set as Dunlearn and the retain set as Pretain = Derain Puuneera (1 denotes removal here); (3) We define two parties involved in unlcarning: the model owner & who receives the unlcarning request, and the data owner who wishes to remove data. In practical scenarios, consist of group of individuals ii=1 who may not know the existence of each other, but participate in unlearning at the same time.
Approximate Unlearning: The model owner E aims at choosing an unlearning algorithm that approxi- mates exact unlearning (training on Drotain from scratch) to obtain g. In the meantime, this algorithm should be much more efficient than exact unlearning. Given a pool of candidate algorithms, & performs white-bor evaluations (access to Derein. Punlearn: Drest. 9 and candidates {ğ)) by comparing their performances with exact unlearning according to metrics in Section 3.3.
Unlearning evaluation: Assuming the unlearning process is finished and the model owner has published the final unlearned model 9, an individual data owner wishes to examine the unlearning outcome. We coin this process as unlearning evaluation. Specifically, such evaluation is black-boz as only has access to his/her own unlearning subset Di Dintesrar and the output of the model before g( and after unlearning @( By comparing these outputs, performed and the output is as desired. Our paper aims to provide such evidence through the design of a novel unlearning algorithm for contrastive learning.
should find explicit evidence that unlearning has indeed been
Note that in practice, 9 may consist. of a general encoder and projection head (which typically removed for downstream tasks)), as in Chen al. (2020). For simplicity, we use g to represent this entire mapping.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Note that making unlearning easy to evaluate is an important and difficult task for approximate unlearning algorithms (Thudi et al., 2022). While we provide easy-to-check unlearning traces in the later sections, such tools are specifically designed for our algorithm in contrastive learning, and may not be generalized to other unlearning scenarios.
3.2 Adapting existing methods to MUC
We first adapt some existing unlearning methods designed for supervised unlearning to contrastive unlearning. Due to the lack of labels in contrastive learning pre-training, some approaches cannot be directly applied. For example, random labeling (Golatkar et al., 2020; Fan et al., 2024) relies on flipping the labels of the unlearn data; boundary unlearning (Chen et al., 2023) expands or shrinks the decision boundary, which does not exist in our context. In contrast, some other unlearning methods can be tailored to contrastive learning. Specifically, we adapt the following methods:
Retraining: (exact unlearning) trains on Dratain from scratch via minimizing Equation (1);
Fine-tuning (Golatkar et al., 2020) updates the pre-trained model for several epochs on Prataini
Gradient Ascent (Golatkar al., 2020; Neel al., 2021; Thudi et al., 2022) reversely maximizes Equation (1) the Dullsarn
NegGrad (Kurmanji et al., 2023) jointly minimizes and maximizes Equation (1) on Dretain and D Dunlearu respectively;
& f1-Sparytty (Jia et al., 2023) regularizes the & -ormr of model parameters based on fine-tuning.
The above methods manifest straightforward adaptations of unlearning from supervised learning to contrastive learning by changing the cross entropy loss to the InfoNCE loss in Equation (1). In Section 5, show that approximate unlcarning methods are suboptimal approximations of exact unlearning, namely that there still exists a performance gap compared with retraining. This motivates us to design new unlearning methods specifically for contrastive learning in Section 4.
3.3 How to choose an unlearning algorithm
Suppose the model owner & gathers a pool of unlearning algorithms (e.g., the methods above). Next we introduce how to compare them, (.e., the evaluation metrics. As contrastive learning returns a feature extractor, we can either evaluate unlearning on the representations directly or rely on downstream tasks. Specifically:
Represeentation-leel metrics. Forgetting Score: given candidate algorithm that updates g to 9, we propose a Forgetting Score (FS) by directly adapting the memorization score in evaluating data attribution in Wang et al. (2024). FS measures the quantity of forgetting D Duniearn by comparing the alignment loss through the features returned by model paramcters before and after unlcarning:
FS
E s(g(z). (a.))~p
E (w.))~p
s(g(z).(u)),
(2)
where Pa is the density of Puntearn, recall g and 9 are models before/afie aater unlearning. @ Membership Inference Attacks (MIA): MIAs are capable of indicating whether certain samples (e.g., the unlearn set) are included in the training set and are perfect for examining unlearning efficacy. EncoderMI (Liu et al., 2021) proposed an alignment-based membership inference attack for self-supervised encoders. It extracts membership information from the embedded features to distinguish whether input data included in the encoder training set. Following the implementation of Jia et al. (2023); Fan et al. (2024), we evaluate the attack success rate (ASR) on the unlearn datasct Duniearn and denote it by encoder membership inference attack (EMIA) efficacy. We compare EMIA on Dunleara with retraining. See Appendix A.3 for details of FMTA.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Downstream-level metrics. Alternatively, representations can be evaluated with downstream tasks. We perform linear probing, i.e,, image classification the same (labeled) dataset for unimodal contrastive learning. Given the unlearned encoder I returned by candidate algorithm, we train an additional linear head on Dretain on top of the fixed 9 to obtain a classifier. Next we evaluate: Accuracies: we evaluate relain accuracy (RA) on Dretain, lest accuracy on Deest (TA), and unlearn accuracy on Dunleamn (UA). For good unlearning algorithm, the above three measurements should be close to those of the retrained model, with a common pattern of UA TA < RA; @ Membership Inference Attacks: Similarly to EMIA, membership inference attack (Jia et al., 2023; Fan et al., 2024; Song &
implement confidence- Mittal, 2021) on the entire network (encoder and linear head) and report classifier membership inference attack (CMIA) efficacy. We compare CMIA with retraining. See Appendix A.3 for details.
3.4 How to evaluate unlearning
After choosing an optimal unlearning algorithm, the model owner generates the unlearned model 9 as a response to the unlcarning request made by data owners. However, it is impossible for the data owners perform the same white-bor bor evaluations.
For the data owners = (Unlearning evaluation) : Recall that an individual data owner performs black-bor evaluation due to the limited access to imput Di Dimtearn and the output of the encoder bcfore/aftor unlearning. Specifically, obtain TA or R.A to quantify performance. Additionally, there is a lack of the retrain baseline to compare with. To this end, the only evaluation tool the forgetting score FS on Duntearm: which can be calculated with Equation (2). However, we argue that this evaluation is neither sufficient nor reliable, and we use simple empirical example to validate this claim:
cannot train shadow models with Dineearn alone to perform MIAs; and cannot
Eract unlearning on MoCo (He et al., 2020): Using a fixed random seed of 11, we pretrain a MoCo encoder 91 on the entire CIFAR-10 training set. For exact unlearning, we retrain an encoder 92 after removing 10% of the data, with the same random seed. To simulate a scenario where the model owner attempts to deceive the unlcarning process, we assume that instcad of retraining, the original encoder 9I is replaced with another encoder g3 that pre-trained on the entire training data with a different random seed of 10. In this setting, the null hypothesis Ho (no unlearning) corresponds to the deceptive case in which 91 is replaced by 93- The alternative hypothesis H1 represents the exact unlearning case where 91 is replaced by 92- For both hypotheses, we calculate the FS for every unlearn sample and get the mean μ and the standard deviation a across the 4500 unlearning images. The resulting statistics are: Hg : (No = - 0.0026, op 0.0587); Hi : (H1 = 0.0353, 0 = 0.0575). Assuming Gaussian distributions and a data owner possesses images, we perform t-test to evaluate the p-value for distinguishing between Hg and H1:
Table 1: p-value of the t-test on distinguishing between hypothesis Ho and Hi,
N
5
10
15
20
p-value
0.3322
0.1617
0.0847
0.0459
In Table 1, we observe that when the data owner holds fewer than 15 images, it becomes difficult to statistically detect that the model owner has cheated the unlearning process.
In summary, under the current. MUC framework, existing approximate unlearning algorithms and evaluation tools are insufficient. To address this, we design new unlearning algorithms for the model owners and advanced evaluation tools for data owners in contrastive learning in the next section, which would benefit both parties in engaging the unlearning procedure.
Surrogate models train classifiers for membership inference.
6

---

Published in Transactions on Machine Learning Research (08/2025)
4 Alignment Calibration
4.1 Tailored objective for MUC
We first introduce more effective unlearner for model owners C. Recall that E 's goal for unlearning is preserving the model utility on Dretain while revoking the effects of training on Dullearn. For the retain dataset Dretain we minimize the InfoNCE loss in Equation (1) to achieve reasonable downstream performance after unlearning:
Cretain
E s(o(x). (x.g))pp
+ E log E exp(s(g(=). 2-ppe y-pp
(3)
where Pr is the density of Dretain and Pa is the density of Derain.
For the unlearn dataset Danlearn revoking the effects of training amounts to achieving the following goals upon evaluation in Section 3.3:
(Encoder-level) Enlarging forgetting Dunleam recall that in Equation (2) the forgetting score FS is measured by the difference between feature similarity Dunlearn before/after unlearning with pre-trained model and unlearned model g. As the first term is fixed (as is given) during unlearning, increasing FS is equal to minimizing the second positive alignment term. For this purpose, we explicitly perform such minimization in our objective function and call it positive alignment calibration.
(Dounstreamm-kevl) UA TA < RA: enlarging FS alone may also hurt the overall downstream perfor- mance on the unlearned model g. To obtain reasonable UA and TA, we find it beneficial to maintain the term for negative pairs in contrastive learning, such that for Dunlaam, we minimize:
Cunlearn
E (z:9))pp
((g()),(())
E log E exp(s(g(#)) prup
(4)
grupa
positivv alignment cakbration
performance preserving
where Pa is the density of Dunlsarn Wang & Tsola (2020) states that alignment and uniformity are crucial properties of good representations. While we calibrate alignment with the first term, our performance preserving term implicitly maintains uniformity, which we will demonstrate in Section 5.5.
4.2 Calibration under unlearning evaluation
Evaluation beyond FS: Recall that in Section 3.3, we show that the forgetting score FS not a sufficient.
nor reliable evaluation for unlcarning success. Here we introduce an additional evaluation tool: given Dunlearn and the models before unlearning 9; data owners + can easily obtain the feature vectors with two different. data augmentations: g(x) = {g(xx)) |Pantam and g(y) = (s(s))) [Paasmal Then an Alignment Matrix : AM(g(x). gyy) can be casily acquired by calculating the pairwise similarity between the two vectors. See Figure 3(a) for some visualizations of AM in the format of heatmaps. Similarly, * can obtain AM(}(x),(y))) after unlearning and an additional Aligument Gap Matrix AGM AM(g gy))) AM()(x). ((y)) The heatmaps of AM and AGM provide evaluation tools beyond FS and allow to visualize the model change through unlearning by looking at the temperature of the graphs. Notably, the elements on the diagonal of AGM also visualize sample-wise forgetting scores. We direct readers to visualizations of such graphs in Figure 2 and Figure& 3 in Section 5.
Taking evaluation into account for unlcarning: The additional evaluation tools enable the model owners to design an algorithm that allows data owners to clearly visualize the effect caused by unlearning (Le., through AM or AGM) without sacrificing the goal of unlearning.
We provide simple solution to improve existing unlcarning methods. As we have explicitly calibrated the alignment of positive pairs of Luitern in Equation (4), it suffices to adjust that of negative pairs (within D Dunlearn) to larger value to enlarge the model differences in AM. Specifically, update the unlearn loss in
7

---

Published in Transactions on Machine Learning Research (08/2025)
Equation (4) with negative alignment calibration:
E
s(g(z),((V))
+
E
s(g(x),(()))
+y- I xrpps
log E exp(s(g(z). grupa
negatiee alignmmnt celibrution
posstioe aliymment colibration
performance preserving
where a, y are tunable parameters adjust the strength of cach component, P represents negative pairs within the negalive set. We write the complete objective of Alignment Calibralion (AC):
Cretain Lunlearn
(5)
where & = Puntearnl/12 // varies by the size of the unlearn set. In the next section, we will show AC' not only achieves state-of-te theaart performance upon model owners' evaluations but can also easily pass data owners' visual ovaluation on unlearning.
5
Experiment
Recall that we made several claims in Section 3 and Section 4: 0 Existing methods are suboptimal unlearners under white-bor evaluations and our AC algorithm approaches exact unlearning in this regard; @ Under MUC, AC introduces alignment matrices AM for visual evaluation and exhibits clear evidence for unlearning. In this section, we evaluate the baseline methods and AC following the above steps and present white-bor evoluation by model owners & and black-bor evaluation by data owners
5.1 Experimental Setup
Data and models. For unimodal castrastive unlearning, we perform experiments CIFAR-10/CIFAR-ION Krizhevsky et al. (2009)/SVHN (Netzer et al., 2011) and SimCLR (Chen et al., /Moo (He et al., 2020) algorithms with the ResNet-18 (He et al., 2016) backbone (we provide additional results on ResNet-50 in Table 12 in Appendix B). We randomly forget 10/50% training data from pre-trained encoder. For multimodal contrastive unlearning, we evaluate CLIP (Radford et al., 2021) on an Image- Text paired dataset called MS-COCO (Lin et al., 2014), which contains 120K images and 600K captions. We perform unlearning on 10% randomly selected image-text pairs.
Whitc-box Evaluation: Following Section 3.3, FS and EMIA for encoder-level evaluation, and use CMIA, RA, TA, and UA for downstream-leve evaluation after performing linear probing for SimCLR MinCLR/MoCo MooCo experiments. For the evaluation of CLIP, we measure the image-text cosine similarity of the retain dataset. and unlearn dataset due to the lack of suitable downstream tasks. Across all experiments, we compare each unlearning method with the exact unlearning (retraining) baseline and report the differences across all metrics. We also report the running time efficiency (RTE) of unlearning methods evaluate efficiency.
Black-box evaluation: Recall that due to the limited access of data owners, the above white-box evaluation not be directly applied. Instead, we use the Al Alignment Matrix (AM) and Alignment Gap Matrix (AGM) introduced in Section 4 for visual evaluation on MoCo and CLIP.
Unlearning Algorithms: We evaluate Retrain, Fine-Turm Gradient Ascent, NegGrad, and L EpSSprsity as baselines for MUC. Our Aligmment Calibration method updates the pre-trained encoder for the same number of epochs as FineTune, NegGrad, and f--Sparsity. For simplicity, we set a = y = 1 if not otherwise stated and we tune 3 for the best. performance. Implementation details of the above methods are described Appendix A.2.
5.2 Unlearning Performance under White- box Evaluation
We first provide empirical evidence for model owners to choose a suitable unlcarning method with superb efficiency and effectiveness. Unimodal contrastive learning: We present our evaluation under EMIA, RA, TA, UA and CMIA in Table 2 and Table 3 for unlearning 10/50% of CIFAR-10 training set and FS (CIFAR-10,
8

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Unlearning performance of different methods on randomly forgetting 10% of CIFAR-10 training data under various metrics. The performance gaps between retraining and other methods are shown the parenthesis. We report the average gap (Avg. Gap) over these 5 metrics. The results are obtained by averaging over 5 random trials.
Methods
EMIA
RA
TA
UA
CMIA
Avg.
Gap (%) J
RTE (mins)
MoCo
Retrain
49.72
89.54
87.76
88,42
34.38
109.47
Fine-Tune
50.15 (0.43)
88.34
(1.20)
86.46 (1.30)
87.59 (0.83)
29.42
(4.96)
1.74
1.42
Grad. Ascent
44.95 (4.77)
89.92 (0.38)
88.28 (0.52)
89.76 (1.34)
28.53 (5.85)
2.67
0.17
NegGrad
48.43 (1.29)
89.25 (0.29)
87.35 (0.40)
88.58 (0.16)
28.89 (5.49)
1.53
1.70
E E--pearity
49.38 (0.34)
88.56 (0.98)
86.91 (0.84)
88.12 (0.30)
29.91
(4.47)
1.39
1.43
AC (Ours)
50.28 (0.56)
89.14 (0.40)
87.24 (0.52)
88, 88.20 (0.22)
31.50 (2.88)
0.92
1.87
SIMLLR
Retrain
48.11
90.87
88.94
89.68
38.87
151.77
Fine-Tane
47.72 (0.39)
89.38 (1.49)
87.26
(1.68)
88.93 (0.75)
30.71 (8.16)
2.49
1.93
Grad. Ascent
41.48 (6.63)
91.26 (0.40)
89.55 (0.61)
91.11 (1.43)
29.36 (9.50)
3.71
0.19
NegGrad
49.56 (1.46)
89.10 (1.77)
87.23 (1.71)
89.07 (0.61)
29.97 (8.89)
2.89
2.34
E SpaSptsiyy
48.44 (0.33)
90.59 (0.28)
88.56 (0.38)
90.44 (0.75)
30.61 (8.26)
2.00
1.96
AC (Ours)
48.64 (0.53)
90.24 (0.63)
88.06 (0.88)
89.24 (0.44)
33.12 (5.75)
1.65
3.00
Table Unlearning performance of various methods on randomly forgetting 50% of CIFAR-10 training data. The results are averaged over 5 random trials.
Methods
EMIA
RA
TA
UA
CMIA
Avg. Gap (%)
RTE (mins) +
MoCo
Retrain
55.95
85.98
83.55
83.98
46.66
66.71
Fine-Tune
49.98 (5.97)
87.89 (1.90)
85.66 (2.12)
86.65 (2.67)
32.35
(14.31)
5.39
0.86
Grad. Ascent
42.90 (13.06)
89.51 (3.53)
87.79 (4.25)
88.99 (5.01)
31.85 (14.82)
8.13
0.45
NegGrad
57.40 (1.45)
83.04 (2.94)
80.15 (3.40)
80.59 (3.39)
40.65 (6.02)
3.44
1.66
E--Sparsity
52.19 (3.76)
81.60 (1.38)
80.19 (3.36)
80.77 (3.20)
38.43 (8.24)
4.59
0.87
AC (Ours)
55.02 (0.93)
86.28 (0.30)
83.28 (0.26)
83.72 (0.26)
38.39 (8.27)
2.00
1.84
SIMCLR
Retrain
53.37
87.23
85.16
85.69
49.30
89.74
Fine-Tune
45.90 (7.47)
87.88 (0.65)
85.50 (0.34)
87.23 (1.54)
35.44 (13.85)
4.77
1.17
Grad. Ascent
42.23 (11.13)
90.52 (3.28)
88.61 (3.45)
90.45 (4.77)
33.28 (16.02)
7.73
0.59
NegGrad
55.70 (2.33)
83.98 (3.25)
82.15 (3.01)
83.80 (1.89)
33.67 (15.62)
5.22
2.32
f1-ppariity
46.51 (6.86)
89.88 89.84 (2.60)
87.75 (2.69)
89.48 (3.80)
35.38 (13.91)
5.95
1.19
AC (Our)
47.12 (6.25)
86.11 (1.13)
83.92 (1.24)
85.24 (0.45)
37.57 (11.72)
4.16
3.07
MoCo) for 10/50% separately in Table 8 due to different scales. We also report the average gap percentage in Appendix B.3. For both SimCLR and MoCo, proposed Alignment Calibration (AC) method achieves the lowest average performance gap over FMTA, RA, TA, UA, and CMTA. In terms of unlearning efficiency, our method only introduces slight overhead. Additionally, our method achieves the lowest FS gap compared to retraining. In Table 4 and Table 5, we also report the results on CIFAR-100, where our methods consistently achieve the best performance. In Table 6, we report the unlearning results on the SVHN dataset. Our Alignment. Calibration method achioves the least average gap to Retraining compared the other bascline methods while showing comparable efficiency in the unlearning process. Multi-modal contrastive learning: in Table again observe that our method is the best approximator of exact unlearning when evaluating the image-text cosine similarity.
5.3 Black-box evaluation
Motivated by the insufficencc of evaluation with the FS score, we propose to apply the Alignment Matrix (AM) and Alignment Gap Matrix (AGM) in Scction AM and AGM naturally introduce additional quantifica- tion of negative alignment. In Figure 2, we report the negative alignment value (mean and standard deviation of pairwise similarity negative samples in AGM) of 4500 unlearn samples and observe that method AC

---

Published in Transactions on Machine Learning Research (08/2025)
Table 4: Unlearning performance of various methods on randomly forgetting 10% of CIFAR-100 training data. The results are averaged over random trials.
Methods
EMIA
RA
TA
UA
CMIA
Avg. Gap (%)
RTE (mins)
MoCo
Retrain
56.24
62.23
58.60
58.43
59.53
109.47
Fine-TTne
46.05 (10.19)
63.49 (1.27)
58.88 (0.29)
59.80 (1.37)
48.81 (10.72)
4.77
1.42
Grad. Ascent
44.01 (12.23)
62.66 (0.33)
59.00 (0.41)
60.65 (2.22)
53.28 (6.25)
3.96
0.17
NegGrad
53.38 (2.66)
63.70 (1.47)
58.78 (0.19)
58.85 (0.42)
48.68 (10.84)
3.12
1.70
f1-Sparsity
45.68 (10.56)
60.89 (1.34)
57.40 (1.20)
58.66 (0.23)
52.48 (7.04)
4.07
1.43
AC (Ours)
50.17 (6.07)
63.20 (0.97)
58.56 (0.04)
58.44 (0.00)
54.15 (5.38)
2.49
1.87
SIMCLR
Retrain
51.20
57.76
56.25
55.86
65.60
151.77
Fine-Tune
40.85 (10.35)
57.29
(0.48)
54.96
(1.29)
55.85 (0.00)
60.61 (4.99)
3.42
1.93
Grad. Ascent
34.00 (17.21)
62.12
(4.36)
59.58 (3.32)
61.08 (5.23)
54.21 (11.39)
8.30
0.19
NegGrad
46.39 (4.81)
56.52 (1.24)
54.30 (1.95)
55.00 (0.86)
60.04 (5.56)
2.89
2.34
Ep-Spariity
40.55 (10.66)
57.85
(0.09)
55.76 (0.49)
56.68 (0.83)
58.46 (7.15)
3.84
1.96
AC (Ours)
46.72 (4.48)
57.11 (0.65)
54.70 (2)
55.23 (0.63)
59.78 (5.83)
2.63
3.00
Table Performance of methods randomly forgetting 50% of CIFAR-100 training data. EMIA is evaluated the unlearned encoder, while RA, TA, UA, and MIA are evaluated after linear probing. We report the average gap (Avg. Gap) over these 5 metrics ween methods and Retrain. The results are averaged over 5 random trials.
Methods
EMIA
RA
TA
UA
CMIA
Avg. Gap
RTE
MoCo
Retrain
60.40
57.72
52.58
52.32
67.30
66. 71
Fine-Tune
53.58 (6.81)
61.9 (4.18)
56.07 (3.48)
56.87
(4.55)
52.27
(15.03)
6.81
0.86
Grad. Ascent
43.76 (16.63)
60.91 (3.19)
56.2 (3.62)
57.48
(5.16)
55.64 (11.66)
8.05
0.45
NegGrad
38.95 (21.44)
60.94 (3.22)
57.01 (4.43)
58.21 (5.89)
57.64 (9.66)
8.93
1.66
fiSSpariity
49.97 (10.43)
58.89 (1.17)
53.07 (0.49)
53.66
(1.33)
56.27 (11.03)
4.89
0.87
AC (Ours)
56.22 (4.18)
59.53 (1.81)
53.37 (0.79)
53.69
(1.37)
52.27 (15.03)
4.63
1.84
SIMCLR
Retrain
56.00
50.40
48.46
47.68
69.47
89.74
Fine-TTuee
53.89 12)
52.82 (2.42)
49.87 (1.41)
51.04
(3.36)
60.06 (9.41)
3.74
1.17
Grad. Ascent
40.28 (15.72)
56.32 (5.92)
54.12 (5.66)
55.22 (7.54)
61.43 (8.04)
8.58
0.59
NegGrad
46.83 (9.17)
50.60 (0.20)
48.20 (0.26)
49.29 (1.62)
58.01 (11.47)
4.54
2.32
f1-Sparsity
45.12 (10.89)
52.62 (2.22)
50.09 (1.62)
50.98 (3.30)
65.25 (4.22)
4.45
1.19
AC (Ours)
54.98 (1.02)
49.28 (1.12)
46.8 (1.66)
47,00 (0.68)
57.78 (11.69)
3.24
3.07
Table 6:
Performance of methods randomly forgetting 10% of SVHN training data from pre-trained MoCo encoder. The results averaged over 5 random trials.
Methods
EMIA
RA
TA
UA
CMIA
Avg. Gap
RTE
Retrain
51.24
91.17
92.02
90.59
20.46
114.13
Fine-Tuie Tune
50.69
(0.54)
90.19 (0.98)
90.63 (1.39)
89.75 (0.84)
17.03 (3.43)
1.44
1.41
Grad. Ascent
46.45 (4.79)
91.23 (0.06)
92.25 (0.22)
90.94 (0.35)
18.44 (2.02)
1.49
0.17
NegGrad
50.99 (0.24)
90.67 (0.50)
91.14 (0.88)
90.08 (0.50)
17.42 (3.03)
1.03
1.74
f--pparsity
50.56 (0.67)
89.20 (1.97)
89.72 (2.31)
88.89 (1.70)
20.52 (0.06)
1.34
1.45
AC (Ours)
51.50 (0.26)
90.19 (0.98)
90.76 (1.27)
89.23 (1.36)
19.99 (0.46)
0.87
1.98
exhibits a more significant unlearning effect under such evaluation. For individual data owners the size of their subset [Di (Dintaamnl may be small. Therefore, we provide additional qualitative results for visual evaluation: we randomly select 8 samples from Dulearm to simulate the budget of We construct AM (before/afi /after unlearning with AC) and AGM for this small set and plot their heatmaps in Figure and observe the apparent.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Performance of methods on randomly forgetting 10% of MS-COCO data from a pre-trained CLIP.
We report the imago-text cosine similarity of the retain dataset and unlearn dataset respectively, as well the average absolute gap from Retrain. The results are averaged over random trials.
Datasct
Pre-train
Retrain
Fine-Tune
Grad. Ascent
NegGrad
4--ppariity
AC (Ours)
Retrain
62.09
62.47 (0)
58.67 (3.80)
61.96 (0.51)
58.57 (3.90)
57.70 (4.76)
60.75 (1.72)
Unlearn
62.08
49.84 (0)
54.75 (4.91)
62.03 (12.19)
49.19 (0.65)
53.54 (3.70)
51.23 (1.39)
Avg. Gap (%)
0
4.35
6.35
2.28
4.23
1.56
Table 8: Forgetting score (FS) of methods for CIFAR-10 and MoCo. gaps are computed be- tween Retrain and other methods.
Figure 2: Negative alignment of 4500 unlearn sam- ples (10%) and MoCo and CIFAR-10. The error bar is the standard deviation.
10%
50%
0.10
Methods
FS
Gap
FS
Gap +
0.05
Fine-Tune
Retrain
0.0266
0.0604
0.00
Grad. Ascent
Fine-Tune
0.0393
0.0127
0.0423
0.0180
NegGrad
Grad.Ascent
0.0005
0.0262
0.0007
0.0596
0.05
It Sparsity
NegGrad
0.0205
0.0061
0010
0.0398
f--Sparsity
0.0216
0.0050
0.0408
0.0195
-0.10
AC (ours)
AC (Ours)
0.0259
0.0007
0.0672
0.0068
-0115
effect of unlearning. We provide additional results for other methods and CLIP unlearning in Figure 6 and Figure/ 7 in Appendix B.2, where AC consistently exhibits the best performance under visual evaluation.
To demonstrate the statistical reliability of negative alignment gap a black-box evaluation metric, we consider using it to distinguish between the null hypothesis Ho where the model owner replaces the encoder (trained with random seed 11) with another one trained using random seed 10, which still contains information about the unlearned data, and the alternative hypothesis Hi where unlearning has been performed using AC with a = 1 =8, and random seed 11. We compute the average negative alignment gap and its standard deviation across the unlearn data in the task of unlearning 10% data from MoCo encoder. The resulting statistics are: Ho (H6 0.0057, a6 0.0403); Hi ( -0.0359, of 0.0295). Note that data owner possessing N unlearn images can obtain N(N-12 negative aligmment values. Assuming Gaussian distributions, we perform a t-test evaluate the p-value for distinguishing between Hj and Hí. We observe from Table 9 that the p-value for N = is sufficiently small such that the data owner can confidently exclude the null hypothesis Hy- Table also shows that t-test with FS fails in distinguishing AC-unlearning and the cheating case (See Appendix B.9 for more details).
2
Pretrain
Alignment Callbration
Gap(Pretrain, AC)
0:87
0.21
0:24
0.3"
0:17
0.15
3.19
0.20
0.86
6:28
0.42
0.37
6:22
0.25
0:24
0.01
4.04
40/77
0.04
4005
4.06
0404
0.21
8.00
5.15
0.39
0.17
2:27
0.25
0.22
8.77
0:22
0/41
0:39
0.30
-2.00
0.05
0.59
8:02
4.00
0:37
0.15
0.04
24
2:23
0.25
0.27
0.22
0.50
0:23
225
0.37
0:32
+3.0 m.
0.07
+0:05
3.05
4
0007
4 0
06
8:17
0:30
0:20
0.24
0
0:21
9:69
0:50
0.22
0:41
26
0.90
0.50
0:39
8:03
9.19
0.37
0.28
8.5
3.02
0.58
0.25
0-20
0.50
0.37
0.90
0.50
0.44
4.005
0.62
-U00
4.00
0.06
-003
0.08
0.66
8:20
0.26
0.25
0.20
3:30
8.
0:24
0:32
0:37
0.14
8788
-1.M
0.50
6:05
-007
4.00
Figure 3: Alignment Matrices and Alignment Gap Matrix on random images in unlearn dataset of CIFAR-10 (MoCo). The task is forgetting 10% of training data.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 9: p-value of the t-test on distinguishing between hypothesis Ho and Hi.
Metric \N
5
10
15
20
FS
0.7452
0.64
0.5648
0.5052
Neg. Align.
Gap
0.0168
<0.0001
$0.0001
$0.0001
Table 10: Ablation study on positive and negative calibration Equation (5) regarding the average gap over metries on CIFAR-10 and MoCo with forgetting ratio 10/50%. "w/" denote with and "w/o" denotes without. For example, "w/o, w/" means AC' without negative calibration but with positive calibration.
Figure 4: The effect of a and 3 on the forget- ting score ratio between Retrain and AC, i.c, FS(RT):FS(AC). Here forget 10% of CIFAR-10 training data from MoCo encoder.
2.5
FS(RT) FS(AC)
1.1
2.0
FS(RT): : FS(ACC
Neg. Cal.
Pos. Cal.
Forgetting 10%
50%
W
w
0.92
2.00
1.0
W/O
1.25
4.12
1.0
W
W
1.95
3.75
W/
W
2.60
6.45
0.9
0.5
0.4
1.2
1.6
16
0
5.4 Ablation Study
Influence of negative alignment calibration: In Equation (5), the coefficient a controls the intensity of maximizing the negative alignment on unlearn data. To explore the effect negative alignment calibration in the unlcarning task, fix 3 and adjust while kceping y for simplicity. Figure 4 (orange bars) reports the ratio between the forgetting score FS of Retrain and AC. When a increases, the ralio decreases, indicating that the resulting model forgets more information about the unlearn data. A ratio of 1 denotes that the FS of AC equals that of Retrain. Furthermore, we consider a more extreme case of a 0, representing no negative calibration. In Table 10, without negative calibration, the average gap over metrics is larger than that of the standard AC by 0.33/2.122 12% (comparing rows "w/o, w/" with "w/, w/") for 10/50% forgetting tasks, suggesting this additional term not only benefits the data owner for unlearn evaluation but also improves the unlearn performance.
Influence positive alignment calibration. Equation (5), the coefficient 3 controls the intensity of minimizing the positive aligument on the unlearn data. In Figure 4 (blue columns), we fix a 7 and vary 3 from 0 to 16. The forgetting score ratio decreases with increasing 3 and approximately reaches 1.0 in the range of [12,16]. In Table 10, the positive alignment calibration term enhances the unlearning performance from 1.95/3.70 3.7% to 0.92/2% (comparing columns "w/, w/o" with "w/, w/") for the 10/50% forgetting tasks regarding the average gap.
5.5 Preserving Uniformity
Finally, validate the function of the performance preserving term in loss function, which forces extracted features to spread uniformly on the hyper-sphere. Following the implementation from Wang & Isola (2020), we visualize the uniformity of features on CIFAR-10 before and after applying our AC unlearning algorithm. Specifically, we train a ResNet-18 encoder mapping images to 2D space using SimCLR and plot the distribution of angles ei. arctan2(r, y) for each point (x, S in Figure We observe that this term implicitly maintains the uniformity of the representation on both the test dataset and the unlearn dataset.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Before AC
B = 0
B 2
8 = 10
Angles
Angles
Angles
Angles
Anglea
(a)
Test da lasse
Angles
Angles
Angles
Angles
(b) Unlearn dataset
Figure 5: Angle distributions on test dataset. and unlearn dataset before and after AC unlearning with different 3 values related to positive alignment calibration intensity. Here, the negative alignment calibration and performance--reseree
term intensities ies are kepl at their default values, i.e., a terms take effect in our AC method. Each chart's x-axis represents "Angles" and the y-axis represents "Counts".
1. Note that when
B = O, only negative aligmment calibration and
6 Conclusion
In this paper, we study the problem of machine unlearning for contrastive learning pre-training (MUC). We establish the foundations this line of study by adapting existing unlearning methods and setting up baseline evaluation metrics, including white-boz evaluation for model owners to choose an optimal unlearning strategy, and black-bor auditing for data owners to examine the effect of unlearning. After identifying the suboptimality of existing unlearning methods and the insufficiency of current auditing tools, we further propose our novel method called Alignment Calibration. Our approach introduces novel unlearning objective function to strategically optimize toward the unlcarning goal and enable straightforward visual auditing. Empirically, our method achieves state-of-the-arl performance on unlearning tasks for both unimodal and multimodal contrastive learning.
Limitations and Future Work: (1) Our paper initializes the study of machine unlearning in self-supervised learning but only considers contrastive learning. We plan to extend our exploration of unlearning towards other SSL methods in the future. (2) Pseudo-label-based (Warnecke et al., 2021) and label-free (Foster et al., 2024a;b) unlearning algorithuss may be integrated into our MUC framework as future enhancements. (3) Our paper focuses on approximate unlearning methods for contrastive learning and does not provide theoretical guarantees (e.g., certification of removal), which represents an interesting direction for future research. (4) While our work focus random forgetting, study the difficulty of removing specific samples is important in the contrastive learning setting. We plan to extend existing studies (e.g., (Zhao et al., 2024)) to contrastive learning in future work. (5) Finally, although we perform t-test to show the advantages of our proposed evaluation tool, i.e., the negative alignment values and AGM, we consider one typical deceptive case for the null hypothesis where the model owner cheats by replacing the encoder with a different one without unlearning. In future work, we will investigate more failure cases where malicious model owners deliberately design cheating methods to deceive the metrics proposed in this paper, and propose mitigation strategies accordingly.
Acknowledgement
YY gratefully acknowledges funding support from NSERC, the Ontario early rescarcher program and the Canada CTFAR AT Chairs program. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. XS gratefully acknowledges funding support by the Stralegic Priority Research Program of CAS Grant XDA0480502, Robotic Al-Scientist Platform of CAS, and NSFC Grant 12288201. This research was also funded by the Deutsche Forscungggmmeischaatt (DFG, German Research Foundation), Project number 550224287.
13

---

Published in Transactions on Machine Learning Research (08/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Lucas Bourtoule, Varum Chandrasckaran, Christopher A Choquette- Chono, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. Tn 2021 IEEF Symposium on Security and Privacy (SP), pp. 141 159. IEEE, 2021. URL 09519428. pdf.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463 480. IEEE, 2015.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearn- ing: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings
of the IEFF/CVV Conference on Computer Vision and Pattern Recognation, pp. 7766-7775, wteecceesssteeet...... thecvf com/
2023. URL http://openaccees.
https//pppppr//heeeeeeeeeeeee _eand__oonrr____ Unlearning_ of _aeels_______________ee_______e_ Nwt.wwwwwwwwttteeeeee Saa__vva_Shhhiaa wwii_Shifting.tee theestthe___p _atee_________0 pap.p..ppp....
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020. URL http://prooceddings mlr, ppess/v119/ /chg/// html.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/ /CFF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-96449, 2021.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in 1lms. arXiv preprint arXiv:2310.0225: 2023. URL https://arxt http://xxxvvvv..aeaee. prrg/bbs/2310 02238.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2024. URL / forum?id=gnOmmmhh@G
information theoretic approach to machine unlearning, arXiv preprint arXiv:2402.01400 2024a. Jack Foster, Kyle Fogarty, Stefan Schoepf, Zack Dugue, Cengiz Öxtireli, and Alexandra Brintrup. An
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Loss-free machine unlearning. arXio preprint arXiv:2402.19308 2024b.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. arXio preprint :xXi::2303.07355 2023.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Adilya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020. URL https://openaccess.. theevf cowp//oteettccc//... hapers/Golattar_Bee Etterall__teelll_mm Btteerral_Suushhieeeeeeeee Sunshine, tte_thhe_____ Net_ herraeess_etteeeeeceeeeeeeorrree Dn_______PPPR_________ 2020_ 11200_aper.ppp pdf..
14

---

Published in Transactions on Machine Learning Research (08/2025)
Jean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advences in Neural Information Processing Systems, 03:21271-21281 2020.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Cortified data removal from machine learning models. arXiv preprint arXit:1911.0003 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and poltern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 16000-16009, 2022.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models, 2023.
Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu, and Min Zhang. Separate the wheat from the chaff: Model deficiency unlearning via paramcter-effcient module operation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 18252 18260, 2024. URL attps://arxitss http///rx.v.oorarbabaaa/a wtgg/bs//300..o../ 08000.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference Artificial Intelligence and Statistics, pp. 2016. PMLR, 2021. URL ttttps:/proceeeddng/ mlr pwwpss/rss///epppppp htt.///////221aaaizaaaa izzialiiizz2a..... addf.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.014 015004 2022. URL httpa://arxiv http://aaiiv..rrabb//aa////
httg//aba2210.... 01504.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: https:/ http:///eeereeeeeeemmmmm mett://orrrmddddjjj/33333
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of sccond-order optimization for Ilm unlcarning. arXiv preprint Xiv: 18239, 2024.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Nupur Kumari, Bingliang Zhang, Sheng-Yu Yu Wang, Eli Shechtman, Richard Zhang, and Jun- ian-aan Zhu. Ablating concepts in lext-to-image dillusion models, 2023.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. In shirth-seventh Conference Neural Information Processing Systems, 2023. URL https: //openreview. www.//oormm.dd-veeii.e...
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 19th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Parl V 13, pp. 740-755. Springer, 2014. URL https:/ http:///ooooaaaeccc////// orr/////atttt........o
Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 2081-2095, 2021. URL https: 1445/346611203484774
acm. wwt.aa......//aaatteteeeeee org/doi/paf/10.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Xiaojun Xu, Yugnang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint arxiv:2402.00 08787, 2024.
Yiwei Lu, Guojun Zhang, Sun Sun, Hongyu Guo, and Yaoliang Yu. f-micl: Understanding and generalizing infonce-based contrastive learning. Transactions on Machine Learning Research, 2023.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121 2024.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a. URL https://openrevaal met/foruuriid-Dozzekttto
Eric Mitchell, Charles Lin, Antoine Bossclut, Christopher D Manning, and Chelsea Finn. Memory-basod model editing at scale. In International Conference on Machine Learning, pp- 15817-15831. PMLR, 2022b.
URL attpp://arxitas httppp/:/aarxx.ooogga/////////// 065200.
Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajeer Malvajerdi. Descent-to-dekee Gradient- based methods for machine unlearning. In Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep leorning and unsuperoised feature learning. Granada, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748 8763. PMTR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019. URL http: //proceedings.. mlr pttpps/r////saaasiiiiii html.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text. models. Advances in Newral Information Processing Systems, 35:25278 25294, 2022. URL
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075-18086, 2021.
Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Tony Chen, and Miao Xu. Label- agnostic forgetting: A supervision-free unlearning in deep models. In The Twelfth Internalional Conference on Learning Representations, 2024. URL https://openrevier wwt..nne/oorrmiiiiiiiiiiiiaerrr
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2615-2632, 2021. URL https: //www..wwwwwwww usenix.
pdf.
Anvith Thudi, Hengrui Jia, Lla Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning, In 31st USENIX Securily Symposium (USENIX Securily 22), pp. 4007-4022, 2022. URL https://www.uset usenix org/ httpa//ee//eelleee/e/----a//-//a
.pdf.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic. Learning Theory. PMT.R, 2021.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Han Zhao, Louis-Philipe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. In ICLR, 2021.
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 1126-4142. PMLR, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint (oXt::800.0377/ 2018.
Tongzhou Wang and Phillip Tsola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In JCML. PMLR, 2020.
Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Bocnisch. Memorization in self-supervised learning improves downstream generalization. In The Twelfth International Conference Learning Represenlations, 2024. URL KSjPaXtxP8.
wttp://wwweee/enmmmmtimm frrmmidd
Zixin Wang, Bing Mi, and Kongyang Chen. Encodermu: Machine unlearning in contrastive learning. In International Conference Security and Privacy in New Computing Environments, pp. 212-223. Springer, 2023.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint XrXiv:2108. 11577, 2021.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL http://peereeeeeeeeeeeeeeeeeeeee NZZB3UGed8.
Dawen Zhang, Pamela Finckenberg- Brommm, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. Xix preprint arXiv:2307.039) 2023a. URL
http://attvv..rr//ba///7.00000 03941.
Fric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.1750) 2023b.
Qiuchen Zhang, Carl Yang, Jian Lou, Li Xiong, et al. Contrastive unlearning: A contrastive approach to machine unlcarning. arXiv preprint arXiv:2401.10458 2024a. URL attps://arxiv.orr/htttt 10458v1.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprinl (xXx:2444.00866 2024b.
Kairan Zhao, Meghdad Kurmanji, George-Octavian Bärbulescu, Eleni Triantafillou, and Peter Triantafilouu. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37:12293-12334 2024.
17

---

Published in Transactions on Machine Learning Research (08/2025)
A Experiment Details
A.1 Datasets
CIFAR-10/100. Both datascts consist of 50K training images and 10K test images. All the images are 32x32 colored. CIFAR-100 has 100 calegories and CIFAR-10 has 10 calegories. In unimodal contrastive learning, the augmentations for training encoders include random resizing and cropping, random grayscale, random color jitter, and horizontal flipping. We split the 50K training images into a validation set of 5K images and training set of 45K images. For example, when the unlearning task is to forget 10% of training data, the unlearn dataset Duntearn has 4.5K images and the relain dataset Dretain has 4.05K images.
SVHN SVHN consists of 73,257 training images and 26,032 test images of 10 categories. All the images are 32x32 colored. We split the 10% of training images into a validation set and the rest into a training set.
MS-COCO. COCO is a large-scale object detection, segmentation, and captioning dataset. Its training set contains 118,287 images and 591,753 captions. Each image has several objects and corresponds to at least 5 captions. Different from unimodal contrastive learning which uses strong augmentations, CLIP employs only resizing, center cropping and horizontal flipping to make images of 224x224 pixcls.
A.2 Contrastive Learning and Unlearning Methods
MoCo and SimCLR: For the pre-trained (clean) models, we train the encoder for 800 epochs using an SGD optimizer with cosine-scheduled learning rate initialized at 0.06, momentum of 0.9, and weight. decay of 0.0005. For unlearning methods: Retrain applies the same training strategy as pre-training; Fine-tuning and NegGrad updates the pre-trained encoder for 10 epochs with a learning rate searched in [0.003, 0.03]; Gradient Ascent updates the pre-trained encoder using reversed stochastic gradient descent for 5 epochs with learning rate searched in [10-0, 10-1; -i-Sparyity applies the learning rate as 0.006 and implements & regularization with a coefficient searched in [10 10- For our Alignment Calibration method, the pre-trained encoder for 10 epochs and search the learning rate in [0.003, 0.03] and the tunable parameter 3 in [0, 20] for different unlearning tasks. If not otherwise stated, we adopt a = 7 1. For simplicity, in our reported results on CIFAR-10/100, we use a learning rate of 0.006 for 10% forgetting, and 0.02 for 50% forgetting. The linear probing stage trains a linear classifier head for 100 epochs using an SGD optimizer with a sosine-scheduled learning rate initialized 1.0, and a momentum of 0.9. The batch size is set as 512 for both encoder and linear head training.
update
CLIP: For the pre-trained (clean) CLIP, we train the model for 35 epochs on 2 NVIDIA RTX 4090 GPUs using an AdamW optimizer with a warm-up cosine-scheduled learning rate initialized at 5e-4 and momentum of 0.9. The total batch size is 256 (128 on each GPU). For unlcarning algorithms: Retrain again applics the same training strategy as pre-training, Fine-tuning update the pre-trained model for 8 epochs with a fixed learning rate searched in [5e-5,5e-4]; NegGrad updates the pre-trained model for 8 epochs with a fixed learning rate searched in [10-5, 10-4]; Gradient Ascent updates the pre-trained model for epochs with a fixed learning rate scarched in [5c-6, 5e-4]; l l1-Sparyity updates the pre-trained model for 8 epochs with a learning rate of 0.0005 and a regularization coefficient searched in [10-9, 10-1). For our Aligmment Calibration method, we update the pre-trained model for 8 epochs with fixed learning rate of 0.0002. We search a = y in [0.5, 1] and / in [0, 1].
A.3 Unlearning Evaluation
CMIA efficacy. Given an unlearned encoder I we execute linear probing on it and denote the whole classifier by f. Following the implementation of Jia et al. (2023); Fan et al. (2024), we evaluate the attack successful rate (ASR) on the unlearn dataset Duntearn a confidence based membership inference attack (Song & Mittal, 2021) to f. The formal definition of CMIA efficacy is given by:
CMIA-Efficacy == TNeHIA
(6)
Dunleara
18

---

Published in Transactions on Machine Learning Research (08/2025)
where T'MMIIA is the number of true negatives predicted by the CMIA attack.
EMIA efficacy. We implement the alignment-based EncoderMI-T attack (Liu et al., 2021) in an adapted white-box selting. Given an unlearned encoder with its relain datasel Dretain and test dalaset Dreet: we denote Daon-nanber = Drest sample a subset Dembar of Dratain such that [Pnan-nenber] = |Dnenborl- For each data in Dnoa-netber and nanber, we first augment it 10 times and compute features of these 10 views via g. Then, we compute the cosine similarity between each pair of features, é.e., 45 pairs, and take the average of these similarity values. Now we get a membership feature dataset and a non-membership feature dataset. whose data points are just scalar values. The EncoderMI-T attack then searches for an optimal threshold to classify membership features and non-membership features. Similar to MIA efficacy, the formal definition of EMIA efficacy is given by:
EMIA-Efficage = TNEMIA
(7)
[Duntearn]
where T TENEIIA is the number of true negatives predicted by the EncoderMI attack.
B Additional Experiments
B.1
Unlearning Performance for More Tasks
We present experiment results on CIFAR-100 in Tables 4 and 5. Across these different tasks, our proposed Alignment Calibration method achieves the lowest average gap compared to the Retrain method.
B.2 More Visual Visualization Results
In Figure 6, we report the AGM of Retrain, Fine- Tune, Gradient Ascent, NegGrad, 41Spriisty on CIFAR-10, as a. complement to Figure 3. For the unlearning task on CLIP, we check the ASM of our AC and other bascline methods in Figure 7.
GoptProtratn, Retrain)
GappPretrats, FineTune)
Gappreetain,, Grad Ascent)
GapiProtrain, NegGrad)
GapiPratrain ain, Sparnity)
Cap(Pratrain, AC)
Figure 6: Alig Alignment Gap Matrices of 8 unlearn images for Retrain, Fine-Tune, Gradient Ascent, NegGrad, f1-Sparsity, and our Alignment Calibration. The unlearning task is to forget 10% of CIFAR-10 training data from a MoCo encoder.
Gap(PT, Retrain)
GPp FT)
Gap(PT, GA)
Gap(PT, NG)
Gap(PT, Sparsity)
Gap(PT, AC)
0:15
3.14
0200
1212
2.00 0.00 0.02 0.0 0.00
5:90
43.66
0:53
0,55
0.05
0.65
22:11
0.03
-0.27
+2.00
-2.00
0.12
4:56
0.30
3.96
2.10
2.11
4029
+2.15
4.04
0.12
2.50
0.58
0.00
3.00 0.00 0.1 .001 801
3.00
0.04
0.01
0.00
0.00
3.04
4.04
6.16
0.02
-0U25
-2.10
2.04
-044
8.22
4.54
4.33
0:36
5.00
0.22
0.29
4.00
4.13
0.22
2.50
0.21
0.58
2101 9.01 0.3 000 4.01 0.01
0.14
9.03
0.52
0.05
0.04
3.50
9.6
0.12
4U27
5.10
-110
4:00
4.44
3.17
0.80
0:48
5.55
0.59
600
0.99
3:17
0.00
5.04
6004
0:03
8:00
0.00
5:00 0.50 0.53 6601 8:11
0.00
2.00
0.05
0.52
0.22
0.04
0.00
6:00
43
4013
001
6.00
0.00
4:29
4.56
4.37
0:24
6.94
3/56
5.00
0.2
0.03
5.56
3.00 0.07
3.01
0.00
0.05
4.00
8.00
4.23
3.01 3.00 0.50 001 0.00
0.00
-2000
0.07
0.53
0.05
5.00
4.02
4.00
0.00
4007
4000
0.02
4.18
4:29
4.27
4.32
0.34
0.11
3.54
5.00
-024
0.11
(201
2.07
0.2
2.00
0.12
0.0
-3.0
3.00
0.12
2.01 3.01 0.03 0.0 0.00
0.00
2.00
0.05
0.12
0.11
4.00
0.03
0.00
4.12
0:2
4005
2.04
2.15
2:20
4.27
0.14
0.24
4440
3.15
Figure 7: Alignment Gap Matrices of 6 unlearn imago-text pairs for Retrain, Fine-Tune (FT), Gradient Ascent (GA), NegGrad (NG), EiSparsity, and Alignmend Calibration (AC). The unlearning task is to forget 10% of MS-COCO training data from a CLIP encoder.
19

---

Published in Transactions on Machine Learning Research (08/2025)
B.3 Comparison using Average Gap Percentage Standard
In previous experiments, we compare our AC and baseline methods using the average gap across multiple metrics. To comprehensively illustrate the advantage of our method, we introduce a different standard for comparison, i.e., average gap percentage (AGP), which averages the percentage of GAP over Retrain across multiple metrics. In Table 11, we perform unlearning tasks forgetting 10%/50% of CIFAR-10 data from ResNet-18 SimCLR/MMCC encoders and our proposed AC method still outperforms baseline methods concerning the AGP standard.
Table 11: Comparison in average gap per- centage (AGP, %) with other baseline meth- ods. AGP averages the percentages of Gap over Retrain, i.e,, MEAN(1 retrain })-
Table 12: Unlearning performance of randomly forgetting 10 % of CIFAR-10 training data from ResNet-50 MoCo encoders. We compare AC with baseline methods in Average Gap (AG, %) and the Average Gap Percentage (AGP, %)
CIFAR-10
10%
50%
EMIA
R.A
CMIA
AG
AGP
MoCo
SimCLR
MoCo
SinCLR
Retrain
53.56
92.10
90.10
50.98
35.41
Tune
3.81
9.85
9.01
Pine-Tune
49.85
91.74
89:64
23.94
2.68
8.07
Goad. Ascent
5.83
8.19
14.06
13.35
Grad. Aurent
30.36
92.36
90.71
92.32
25.35
4.43
11.49
NegCrad
3.91
9.10
NegCrad
52.62
91.85
89.78
90.64
24.88
2.07
6.51
f S-SSprsity
3.22
4.70
7-46
10.31
f, SpSpariity
47.61
90.54
88.67
90.19
27. 04
3.63
7.79
AC (oure)
2.16
3.61
4.07
7.75
AC (ours)
53.48
91.75
89.32
89.82
30.56
1.45
3.29
B.4 Model Architecture
In Table 12, we perform an unlearning task of forgetting 10% of CIFAR-10 data from ResNet-50 MoCo encoder. Our AC method outperforms baseline methods concerning both average gap and average gap percentage standards.
Table 13: Standard deviation for Tables 2 to 4. Appendix B.5 shows the detailed settings for random trials.
EMIA
RA
UA
CMIA
EMIA
RA
TA
UA
CMIA
CIFAR- 10,
10%, MoCoo
CIPAR-10, 50%
MoCo
Retrain
18.7226.08
89.54:20.07
87. 87.76220.25
42.00000
31.3820.33
53.9511158
85,9820.25
83.3520027
83,98=0.21
16.0620.77
Fine-Tuu
50.15:17.9
88 14..27
86 86.46.10.1
87.50.10.18
29. 29.42.1..87
49.9811.92
87.89
85.6610.43
32.35 06.65_0.29
Grad, Ascent
11.95=2..1
89 92:10.1
28.22
89.76220.4 46
28 26.53
12.9025.48
89.51
87. 87.7920.24
88 09.19
31. 85=0611
NegGrad
48.4222.42
89. 25:20.24
87.35:2022
88.58110.0 49
28. 20.00.0.66
67.40213.24
83.0420.47
80.1520071
80.5800.66
40.65-1.02
6 -ppariit
49.38.1711
88.56.1o.88
86.91.1o.4
88.12.10.0
29.91 1o.92
52.1911671
L1.60
10.19.00.1
80.77
38, 68.44
AC (Ours)
50.2825.86
89.1122.119
87.21
88, 28..0220.50
31.50:11117
55.0243.114
86.2820.35
$3.28:20.77
83. 72-0.20
38, 19.17
CIFAR-10,
10, 10%, SimCLR
CIFAR-10,
10, 50% SimCER
Retrain
48.11.1470
10.87
88 06.
89.68
35.87.11.85
53.37.12221
87.23.0.25
85. 16.1044
0.11
49.30-_066
Fine-Tune
17. 4.38
89. 39.31100.1
88,
30.7111.61
45.90128.2
87.8820.34
85. 50.0044
87. 23:0000
11=1500
Grad, Ascent
11.18:11611
91220.0111
89.55:20.07
91. 01.11:10.6
29.36110.99
42, 42.23:14.50
90. 90.5220.2
88.6120.15
90. 90.45-0.166
33. 28.48722.4
NegGrad
19.56111446
89.10.10.32
07.23.10.45
89.07 10.48
29.97 1.133
13.70
83.98
82, 15.1.
33.67 03.80
f Sparsity
48 18.44.16.70
00.50-10.23
88 1
0.24
30.61 1.00
46.51 1:3.5
19.14.1..12
87. 07.75-1-.22
69.48 0.22
35. 1.59
AC (Ours)
18.64119.47
20.21:20111
08.00622..22
89.24110.8
33.1211.54
47.1245.84
16.11.:0.0
03.92200.0
85, $5.24=0.60
37..571.116
CIFAR-100.
10%. MoCo
CIFAR-
100, 10%, SimCLR
Retrain
26.21
62. 23:20.27
58. 60:0000222
58, 13:20.04.4 64
19.53
51.20:22007
07.7622.30
56. 25:2..1
55.86=0079
65. 13.60=1.70
Fine-Tune
46.0652-496
63. 19:20.6
58, 88.20.444
59.80 09.
1.111:11
10.85:23.81
57.29 37.2922013
54, 96.6....4
53.85 1..85
60.6114.70
Grud. Ascent
44.01.1270
62. 0111
59.0001..066
60.65.10.64
53.28 1:2
34.001...11
62, 12.12....2
59. 58..22
61.08
54. 54.21-1.91
GegGrad
53.58 4.82
6..70-10.25
58.78.101 0.22
0..55
48.68
46.39
56.52 1-0.5
5..30
55.00 1.08
60.04
& Sparsity
15.6822730
60.89:20.41
57. 47:400:2
68.66.27.78
32.88:2::
40.35::3.43
37.85
56.76.0.34
56.68800.64
58, 100110
AC (Ouns)
10.17
63.20.11..5
1o.56
58.44.10.22
54.15.1..66
46.72.15.50
57. Lo.38
54. 10...6
$5.233....0
59.78
CIFAR-100,
50%, MoCo
CIFAR-100. 1000,
50% SimCLR
Fine-Tune
53.58223.92
61 00:500
56.07720.44
56.87:10.1
32 32.27:10.02
53.89119.87
52.8220.2 29
19.87:0.46
51.04=0.35
60.0662222
Grad. Ascent
43.76.13.62
60.91:10.14
56.20110.18
57.48.10.a1
55.64110.5
40.2811..77
56.32.1..0
54.12.10.5
55.22_0.88
GegGrad
38.95
60.44
57.01
58.21 1.335
1..6
46.8331..95
00.60
48.201.0.45
49.29
58.01 1.90
1y S-Spamity
49.97:24.97
58. 89120.47
63.0720.07
56.27-1100
15:12:95.9
52, 12.66220.53
50.0912079
50.98_0.56
65. 03.25_22.01
AC (Ours)
56.22
59.50 00.73
53.37.10.76
53.60.10.32
52.27.14.06
54.98.12.85
49.28.L0.57
80..01
47.00_0.50
57. 47.78-44.66
B.5 Random Trial Settings
Recall that our results in Tables 2 to 4 are averaged results over five random trials for each unlearning setting. Specifically, we use random seeds 0, 1, 2, 3, and 4. Each random seed determines both the selection of unlearning samples and the corresponding pre-trained and retrained encoders. For instance, with a specific
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 14: Average gaps calculated only cross classifier-level metrics, i.e. RA, TA, UA, and CMIA. The puper-parameters for unlearning methods are chosen optimally regarding this criterion. Bold font denotes the lowest average gap and italic font denotes the second lowest. The average standard deviation across the four metrics is shown in square brackets.
CIFAR-10
CIFAR-100
10%
50%
10%
50%
MoCo
SimCLR
MoCo
SimCLR
MoCo
SimCLR
MoCo
SimCLR
Fine'T'ne
1.69 [0.52]
2.37 [0.57]
5.25 [0.54]
3.53 [0.59]
2.85 [0.73]
1.69 [0.75]
6.44 [0.31]
3.64 [0.83]
Grad. Ascent
1.92 [0:39]
2.98
[0.46]
6.90 [0.32]
6.88 [0.77]
2.30 [0.89]
6.07 [1.33]
5.80 [0.47]
6.79 [1.01]
NegGrad
1.59 [0.46]
2.79 [0.11]
3.58 [0.52]
4.64 [0.87]
3.00 [0.90]
2.14 [1.62]
3.51 [0.99]
3.39 [0.84]
E-- Sy-Sparitty
1.65 [0.51]
2.39
[0.31]
4.80 [2.24]
4.76
[0.74]
2.31 [1.01]
2.07
[1.56]
5.90 [0.55]
2.84
[1.07]
AC (Ours)
0.91 [0.46]
1.89
[0.28]
2.20
[0.45]
3.64
[0.59]
1.60 [0.76]
1.54 [1.46]
4.44
[0.59]
3.87
[1.37]
random seed, our AC method and other baseline methods start from the same pre-trained encoder forget the same samples and aim to approximate the same retrained encoder. The metrics are computed in each random trial. We report the standard derivation for Tables 2 in Table 13.
We observe that EMTA results demonstrate significant instability with respect to random initialization, exhibiting large variances across all bascline methods as well as our proposed approach. To address potential concerns regarding the reliability of this evaluation metric, we conduct additional assessments using only classifier-level metrics (RA, TA, UA, and CMIA). For this evaluation, we carefully select hyperparameters for all unlcarning methods according these criteria to ensure optimal performance and report the results Table 14. The results consistently show that our method still achieves either first second- best performance across all tasks, while maintaining low standard deviation. This substantiates the robustness of our approach even when evaluated through alternative, potentially more reliable metrics.
B.6
Time and Memory Consumption
We acknowledge the rumning time trade-off introduced by AC. To further examine the practicality of AC, we provide additional results in Table 15, which provides detailed runtime and GPU memory usage across models ranging from ResNet-18 ResNet-152. Our results demonstrate that AC introduces minimal computational overhead even for the largest models tested.
Table 15: Comparison time and GPU memory consumption across networks of different sizes. Experiments are conducted on server with four NVIDIA A100 GPUs. GPU Memory is monitored by the Wandb logger.
Time (minutes)
GPU Memory (GB)
RN-18
RN-34
RN-50
RN-101
RN-152
RN-18
RN-34
RN-50
RN-101
RN-152
Retrain
78.09
127.44
243.95
384.44
543.46
5.19
6.91
17.89
24.56
32.62
Fine-Tune
1.03
1.62
3.11
4.89
6.90
5.19
6.91
17.89
24.56
32.62
Grad. Ascent
0.14
0.14
0.25
0.36
0.47
5.19
6.91
17.89
24.56
32.62
NegGrad
1.22
1.96
3.77
5.92
8.33
5.19
6.91
17.89
24.56
32,62
l--pparsity
1.04
1.65
3.15
4.91
6.99
5.19
6.91
17.89
24.56
32.63
AC (Ours)
1.33
2.18
4.16
6.55
9.19
5.21
6.93
17.91
24.58
32.62
B.7 Privacy concerns
We acknowledge that from a differential privacy perspective, visible changes from AC could theoretically enable membership inference attacks by observing differences model behavior compared to retraining (Sckhari et al., 2021), as validated in our AGM analysis in Figure 8. However, we argue that such privacy leakage poses minimal practical risk because:
21

---

Published in Transactions on Machine Learning Research (08/2025)
Limited data access: AGM calculations require access to the private unlearn data, which is typically only available to data owners and model developers;
No baseline acoilabilily: In practice, the relraining process is not performed and retrained models are not released, eliminating the comparison baseline needed for inference;
Restricted adversory capabilities: An adversary would need simultaneous access to both the unlearned model and reference data/models, models, which is unlikely in real deployment scenarios
summary, while the theoretical privacy concern is valid, the practical barriers to exploitation make such risks negligible in realistic threat models where adversarics lack access the necessary private information for meaningful membership inference. Gap(AC, Retrain)
0.02
0.08
0.12
0.04
0.02
0.08
0.03
0.04
0.10
0.05
.0.03
0.04
0.04
0.00
0.07
0.02
0.03
0.12
0.04
-0.01
0.08
0.05
0.08
0.07
0.07
0.04
0.04
0.06
-0.03
0.03
0.06
0.11
0.01
0.05
0.92
0.00
0.05
0.03
-0.01
0.37
-0.01
0.07
0.08
0.06
0.07
-0.04
0.07
0.04
-0.01
0.05
Figure 8: The Gap metric between Retrain and AC on 8 random images in unlearn dataset of CIFAR-10 (MoCo). The task is forgetting 10% of training data.
B.8 More Ablation Studies
Understanding the trade-off between forgetting efficacy and utility preservation crucial for practical unlearning applications. Figure complements Figure by showing how tuning a and / affects utility metrics. This provides complete picture of the forgetting-utin trade-off landscape across different hyperparameter settings.
1.05
RAIRT): ((ART):RAAAE RAJAA)
TAIRT) TA(AC)
UAIRT): :ITT)AAAY UA(AC)
1.00
1.00
0.95
0
0.4
00
1.0
12
0.95
04
0.8
1.0
12
16
0.95
0
0.6
0.8
1.2
12
a
B
B
(a) RA
(b) TA
(c) UA
Figure 9: The effect of and on RA, TA, and UA between Retrain and AC, i.e., FS(RT) ES(RT) FS(C).. Here forget 10% of CIFAR- 10 training data from MoCo encoder.
22

---

Published in Transactions on Machine Learning Research (08/2025)
B.9 More Details Test
Here we clarily the detailed statistios for the I-lest with FS in Section 5.3: the null hypothesis (cheating case) Ho has the same mean and standard deviation (po -0.0026, do = 0.0587) as shown in Section 3.4, and the alternative hypothesis (AC-unlearnig case) Hi has (n" = 0.0084, of = 0.04438).
23


Reviews:
[{"review_id": "rhI1SUfllG", "reviewer": "Reviewer_PgdP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper (up to the results section) is overall well written, and the motiviation and approach is clearly outlined. The proposed framework is targeted at contrastive learning in general and in that sense could be quite impactful in the real-world. At the same time, it is intuitive and easy to apply (assuming access to the training dataset).\n\nThe biggest weakness of this paper is the depth of the empirical analysis, as well as the mismatch between claims about evaluation and what the authors actually did. Broadly speaking, in Section 3.1 the setup is nicely introduced as a process between model owner and data owner, metrics are introduced (and criticised) but then in the results, the authors do not deliver much alongside this story.\n\nI believe this is mostly an issue with writing and result presentation, and less about running fundamentally new experiments --- I think this part is done well.\n\nIn general, the most important result I anticipated while reading everything up to the results is that the data owner would be able to get a model unlearned with baselines and the new AC method, and could then evaluate if the model successfully unlearned the task. A setup for this could be some kind of hypothesis test where the null hypothesis is that the model behaves like the original model, and H0 needs to be rejected with at some p-value to denote that the given model indeed unlearned the task.\n\nThe way the existing metrics are critized (\"Exact unlearning on MoCo\") alludes to this.\n\nInstead, the authors evaluate their new model on a set of metrics and show \"continuous\" improvements over previous methods. From the data owner perspective, \"[the data owner] should find explicit evidence that unlearning has indeed been performed and the output is as desired\"; from the metrics presented, it is unclear to me what the contribution towards this goal is. Does one of the \"new\" metrics perform better than the \"old\" (e.g. RS metric) towards this goal (i.e., is the ranking of algorithms different)? How would the data owner verify that the unlearning was indeed successful?\n\n### Claims & Evidence\n\nLooking at the claims in the abstract,\n\n1. Alignment calibration achieves SOTA\n2. Alignment calibration approximates exact unlearning\n3. Enables visualization of unlearning effects through black-box evaluation\n\nI think that (1) is sufficiently addressed. The authors show various metrics and achieve good results on these metrics. On the flip-side, I do not see the claim for (2) and (3), but might miss something. Could you concisely outline which experimental results support (2) and (3)? Please see the requested changes for additional comments on this (from Q12 onwards).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies machine *un*learning in the context of contrastive learning models. The goal is to take a machine learning model trained using a contrastive learning algorithm like MoCo, SimCLR, or CLIP, and remove knowledge about a subset of its training data from the model. The exact solution to consider for this task is to re-train the full model on the original dataset with the unlearning samples excluded. The authors study the problem of designing approximate unlearning methods that reach the properties of such an exact model with considerable less compute, and discuss how to evaluate such models.\n\nThe central idea is to adopt the contrastive learning objective, which can be split into an alignment and uniformity part (Wang & Isola). On the \"retain\" portion of the data, the CL objective is minimized as before. Added to this is an adapted CL objective where an additional alignment loss with flipped sign is introduced. A weighted sum of both losses is minimized.\n\nThe authors then study the performance of this method against various baselines on CIFAR-10, -100, SVHN for unimodal, and MS-COCO for CLIP.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would advise to re-work the results section. Right now, the authors present a large amount of tables and plots with various metrics, without much insightful interpretation in the text. The main story is along the way of beating the state of the art on metrics the authors previously critizied as not meaningful. I see an opportunity here to actually deliver on the motivaton outlined before, e.g., showing that with MUC, it is actually possible for the data owner to meaningfully verify the unlearning procedure. When re-writing, I would keep this in mind.\n\nAs an alternative option to some of the points, the authors might opt to write an extensive Limitations section, and re-write some of their earlier motivation.\n\nPlease find a list of comments, questions and suggestions below:\n\n1. p.2, \"Unfortunately, direct adaptations of existing unlearning appraoches are unsatisfactory [...]\": Is this an insight from the literature? If so, which paper(s)? If not, where is this shown in the paper later on?\n2. p.3 \"Contrastive Learning\", \"(usually without labels)\": I would not enter this debate here; especially the CLIP example is essentially on paired data, I would advise to simply drop the remark, it does not change the story\n3. $p^x$ before Eq. 1 is not used in Eq. (1)\n4. Eq. (1) omits the fact that many models, e.g. SimCLR, train both an encoder and some kind of projection head on top. This could be minimally noted in the text in case the equation should be kept simple.\n5. p.6., **\"Exact unlearning on MoCO\"**: The paragraph is very anecdotal. Could you transform this into a result/forward reference it? I think it is quite an important statement to make, and I would evaluate this statement properly with a full experiment which shows that FS fails to distinguish the exact unlearned and base models. If this is the case, it would also essentially show that this metric is not useful at all in the further evaluation, but e.g. Table 7 still shows an evaluation (incl. bolded numbers) on it.\n    - As an actionable proposal, a convincing argument would be to consider the RS metric between the baseline and re-trained model, compute a distribution, and test the null hypothesis: mean(RS)=0. This could be a very convincing part of the motivation. I would encourage the authors to look into this and make the statement statistically more precise.\n6. Eq. 3 and 4, is it correct that the negatives are sampled across the whole distribution $p_d$ (vs. $p_r$)?\n7. Eq. 4 the terms of \"positive alignment\" and \"negative alignment\" are a bit confusing. With this naming, the \"positive alignment\" is actually *not* the term that aligns the positive samples (in the sense of Wang & Isola), while the \"negative alignment\" is. I would re-consider the naming here.\n8. The Eq. in \"Taking evaluation into account\" is not labeled; it is also every simple to confuse $p_u^x$ and $p_u^+$ when reading the equation. Could you also clarify what the difference between $p_u^x(x,y)$ and $p_u(x)p_d(y)$ is?\n9. p.7. \"randomly forget\": Did you study what happens when the forgetting request is more targeted, e.g. dropping out a part of a whole class, searching for a particular keyword, removing images associated to a particular photographer?\n10. p.8, \"we tune $\\beta$ for the best performances\" -> how is this tuning done in practice, what is the metric?\n11. Table 1: Where is \"Alignment Calibration (AC)\"? In the text it is mentioned, in the table not. Is that \"Ours\"?\n12. The abstract claims, \"Alignment calibration approximates exact unlearning\". While Table 1 and Table 4 show improved performance, it is unclear why AC approximates this \"more\" than the baselines. Are any claims possible in which fraction of cases (%) you are able to correctly certify that unlearning was successful (= the model becomes indistinguishable from the retrain model)?\n13. What is the standard deviation of the results in the table? Especially with the point raised in Q6, it seems important to report the confidence intervals/standard errors in the table. I would consider re-layouting, and removing the (...) values in favor of standard deviation. A statistical test would be a plus.\n14. FS and downstream metrics are (I think) never compared directly. The abstract claims \"Enables visualization of unlearning effects through black-box evaluation\" --- where is this done/discussed?\n15. Figure 2, Fine-tune, NegGrad, l1, AC essentially looks like the same distribution. Still the text claims, \"exhibits a more signficant unlearning effect\". Can you either run a statistical test showing this, or drop the claim? I think the figure essentially shows what you alluded to in the text: Standard deviation of this statistic is so large that it is essentially impossible to get a meaningful message out of it with this evaluation protocol.\n16. Figure 4 should be converted to a different plot type, or the y axis should start at 0. I would recommend a boxplot w/ error bars or stripplot, or use a table.\n17. Table 8 could be made clearer, there is enough space to put labels above the \"w/ + w/\" column for either readability", "broader_impact_concerns": "The paper presents a method which claims to evaluate the performance of unlearning, which is a legally relevant topic. This makes it interesting and high-impact as a contribution, but also mandates to accurately reflect on the limitations of the algorithms. In particular, I would request the authors to add a section of failure cases of their proposed method, and how to diagnose the respective metrics. I outlined some if these suggestions already above.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M2fllOgwBx", "reviewer": "Reviewer_ownh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Well written and visually pleasing paper\n2. Paper is likely to be of interest to the unlearning community\n3. Good ablation studies to evaluate the proposed method deeply.\n4. Contributions beyond simply another unlearning method has broader value to the community. For example I appreciate the formalization of the two-party model (model owner vs data owner).\n5. Considerations for both white and black-box evaluations is important and valuable\n\nWeaknesses:\n1. I would argue that trying to make the forgetting more visible, via negative-alignment and the AGM, is undesirable. While I appreciate the authors' goal is to provide black-box verification, visible unlearning leaks that the user's information was once in the training data. Sekhari [1] states the goal of unlearning is that with high probability, an observer cannot differentiate between the two cases (i) the model is trained on the set Dt and then a set Du is deleted by the unlearning algorithm and (ii) the model is trained on the set Dr and no points are deleted thereafter by the unlearning algorithm. The proposed negative alignment may violate this, and existing methods that aim to minimise information leakage will likely break the proposed evaluation approach. As a positive, AGM could also be an interesting metric to evaluate this property in the contrastive setting.\n2. The proposed method does have a speed trade-off for a relatively marginal performance gain.\n3. Missing some related literature on unlearning for contrastive models (e.g. [2]). Further, I think there are other methods that may generalise well to the contrastive setting [3,4,5]. [4] could  use positive pairs as pseudo-labels, and [3,5] are both label-free and would work out of the box.\n4. While not a major issue, tuning 3 hyper-parameters is a demanding task.\n\n[1] Sekhari, Ayush, et al. \"Remember what you want to forget: Algorithms for machine unlearning.\" Advances in Neural Information Processing Systems 34 (2021): 18075-18086.\n\n[2]. Wang, Zixin, Bing Mi, and Kongyang Chen. \"EncoderMU: Machine Unlearning in Contrastive Learning.\" International Conference on Security and Privacy in New Computing Environments. Cham: Springer Nature Switzerland, 2023.\n\n[3]. Foster, Jack, Stefan Schoepf, and Alexandra Brintrup. \"Loss-free machine unlearning.\" ICLR Tiny Paper (2024).\n\n[4]. Warnecke, Alexander, et al. \"Machine unlearning of features and labels.\" Network and Distributed System Security Symposium (NDSS) (2023).\n\n[5]. Foster, Jack, et al. An Information Theoretic Approach to Machine Unlearning. Transactions on Machine Learning Research, 2025, https://openreview.net/forum?id=t1utIThKHD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces addresses the problem of unlearning for models trained with contrastive learning. First, a novel unlearning algorithm (AC) is introduced, which is an extension of the InfoNCE loss. Second, Forgetting Score is adapted to the contrastive setting, and also introduces the notion of an Alignment Matrix and Alignment Gap Matrix, to evaluate the divergence in alignment between augmented outputs before/after unlearning.\n\nThe proposed method achieves state of the art performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses section.  I would specifically like to have a discussion on weakness [1], and it would be good to see some updates to literature and/or comparison methods.\n\nOne additional minor correction to the abstract: Contrastive learning methods are not ML models as the author's state: \"While existing approaches address unlearning for classification and generative models, they overlook an important category of machine learning models: contrastive learning (CL) methods\"", "broader_impact_concerns": "No concerns for broader impact at this time.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w6K973ZAsV", "reviewer": "Reviewer_HyuW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Timeliness and Importance: Addresses the critical and increasingly relevant problem of machine unlearning, specifically within the popular and powerful framework of contrastive self-supervised learning.\n\nClear Problem Definition: Effectively delineates the MUC problem, considering the different perspectives and constraints of model owners and data owners (white-box vs. black-box).\n\nNovel and Well-Motivated Algorithm (AC): The proposed Alignment Calibration method is novel and its objective function components are well-justified, directly targeting the goals of forgetting specific data, retaining general performance, and improving evaluability. The concepts of positive/negative alignment calibration are intuitive.\n\n## weakness\nHyperparameter Sensitivity: The AC objective introduces several hyperparameters ($\\alpha, \\beta, \\gamma, \\epsilon$). While some analysis is presented (Figure 4, Table 8), a more thorough sensitivity analysis across different datasets/models regarding their impact on the trade-off between forgetting metrics (FS, MIAs) and utility metrics (RA, TA) would strengthen the work. How crucial is precise tuning for achieving good results?\n\nScalability Concerns: While efficiency (RTE) relative to retraining is shown, the absolute computational cost and memory requirements, especially compared to lightweight methods like Fine-tuning or Grad Ascent, could be discussed further. How does AC scale to significantly larger models (beyond ResNet-50 mentioned in Appendix) or datasets?\n\nTheoretical Justification: The paper provides strong empirical results and intuition, but lacks deeper theoretical analysis. For instance, are there any theoretical insights or bounds on how well AC approximates exact unlearning in terms of information removal or distribution shift compared to retraining?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission addresses the problem of Machine Unlearning for Contrastive Learning (MUC), focusing on efficiently removing the influence of specific data points (Dunlearn) from pre-trained contrastive models (like SimCLR, MoCo, CLIP) without costly retraining from scratch on the remaining data (Dretain).\n\nThe work clearly defines the MUC setting, distinguishing between the needs and capabilities of model owners (white-box evaluation) and data owners (black-box evaluation). It systematically adapts existing supervised unlearning methods (Fine-tuning, Gradient Ascent, NegGrad, $\\ell_1$-Sparsity) to the contrastive learning paradigm (using InfoNCE loss) and establishes their performance as baselines, demonstrating their sub-optimality compared to exact retraining.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discuss Scalability in More Detail: Add a paragraph discussing potential scalability challenges and computational/memory overhead compared to baselines, especially for very large models or datasets. If possible, provide more detailed profiling beyond RTE (e.g., wall-clock time comparison to Fine-Tune).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)