Selective Unlearning via Representation Erasure Using Domain Adversarial Training
Authors: Nazanin Mohammadi Sepahvand
Eleni Triantafillou
Hugo Larochelle
Doina Precup
James J. Clark
Daniel M. Roy
Gintare Karolina Dziugaite
Venue: ICLR 2025 Poster (2025)
Keywords: approximate unlearning
adversarial training
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: When deploying machine learning models in the real world,  we often face the challenge of “unlearning” specific data points or subsets after training.  Inspired by Domain-Adversarial Training of Neural Networks (DANN), we propose a novel algorithm,SURE, for targeted unlearning.SURE treats the process as a domain adaptation problem, where the “forget set” (data to be removed) and a validation set from the same distribution form two distinct domains. We train a domain classifier to discriminate between representations from the forget and validation sets.Using a gradient reversal strategy similar to DANN, we perform gradient updates to the representations to “fool” the domain classifier and thus obfuscate representations belonging to the forget set. Simultaneously, gradient descent is applied to the retain set (original training data minus the forget set) to preserve its classification performance.  Unlike other unlearning approaches whose training objectives are built based on model outputs, SURE directly manipulates the representations.This is key to ensure robustness against a set of more powerful attacks than currently considered in the literature,  that aim to detect which examples were unlearned through access to learned embeddings.  Our thorough experiments reveal that SURE has a better unlearning quality to utility trade-off compared to other standard unlearning techniques for deep neural networks.

Published as a conference paper at ICLR 2025
SELECTIVE UNLEARNING VIA REPRESENTATION ERA- SURE USING DOMAIN ADVERSARIAL TRAINING
Nazanin Mohammadi Sepahvand Eleni Triantafillou Hugo Larochelle Doina Precup 1:2
James J. Clark' Daniel M. Roy Gintare Karolina Dziugaite
McGill University, Mila, Canada
2Googe DeepMind
University of Toronto, Canada
ABSTRACT
When deploying machine learning models in the real world, often face the challenge of "unlearning" specific data points or subsets after training. Inspired by AoveaiaAdversarial Training of Neural Networks (DANN), we propose novel algorithm, SURE, for targeted unlearning. SURE treats the process as a domain adaptation problem, where the "forget set" (data to be removed) and validation set from the same distribution form two distinct domains. We train a domain clas- sifier discriminate between representations from the forget and validation sets. Using gradient reversal strategy similar to DANN, we perform gradient updates the representations to "fool" the domain classifier and thus obfuscate represen- tations belonging to the forget set. Simultaneously, gradient descent is applied to the retain set (original training data minus the forget set) to preserve its classifica- tion performance. Unlike other unlearning approaches whose training objectives are built based on model outputs, SURE directly manipulates the representations. This is key ensure robustness against set of more powerful attacks than cur- rently considered in the literature, that aim to detect which examples were un- learned through access to learned embeddings. Our thorough experiments reveal that SURE has a better unlearning quality to utility trade-off compared to other standard unlearning techniques for deep neural networks.
INTRODUCTION
In machine learning, the principle of unlearning is essential for selectively removing the influence of specific data points from trained model. The gold standard is exact unlearning, which demands that the model behaves as if never encountered the data to be forgotten (the "forget set"). Exact unlearning, however, presents significant challenges in deep learning models due their non-linear exact unlearning may require retraining the model
learning dynamics. A naive implementation on subset of the data that excludes the forget set, which is often prohibitively costly. As result, the focus has shifted towards approximate unlearning methods, which aim to mimic the behavior of oracle model model retrained from scratch without the forget set.
Traditional approximate unlearning methods primarily focus on aligning the model's output after unlearning with that of the oracle model. However, we hypothesize, and show empirically, that it may be possible for an unlearning algorithm to appear to have forgotten, if considering only information in the output space, while information about the forget set may fact remain hidden in intermediate layers and may be recoverable. This is an important setting to consider in case an attacker gains white-box access to the model weights. Motivated by this observation, we study algorithms and metrics that operate directly in the representation space. Interestingly, by visualizing the embedding space (where the model represents data internally), we fact observe that successful unlearning is often correlated with representations closely resembling the oracle's. s, suggesting that directly manipulating representations during unlearning is also a promising approach for designing improved algorithms. Conversely, mismatch in representations can potentially expose the model to a type membership inference attacks (MIAs), which aim to identify whether a specific data point was part of the training data.
"Correspondence sepahvandmiila.queee,,, gkd7 @ @gongle.coommmmmmm

---

Published as a conference paper at ICLR 2025
Motivated by these findings, introduce SURE (Selective Unlearning via Representation Erasure), a novel unlearning method that shifts the focus from the output space to the representation space. SURE aims to adjust the model's representations to closely match those of the oracle, effectively erasing the influence of the forget set.
Through extensive empirical evaluation, demonstrate that SURE not only achieves a superior trade-off between unlearning effectiveness and model utility but also exhibits reduced vulnerability representation-space MIAs.
Contributions.
Enhanced Unlearning Quality: SURE demonstrates superior unlearning quality as as- sessed by standard Membership Inference Attacks (MIAs) and the accuracy difference be- tween the unlearned model and the oracle model. This is further corroborated by comparing the accuracy of unlearned and oracle models on retain, forget, and test sets across various unlearning scenarios, including random, partial class, and class unlearning, as well as sce- narios with a forget set containing outliers from different distributions.
Addressing significant limitation of existing un- learning methods, SURE offers enhanced stability. Unlike methods where minor changes in hyperparameters or forget set size can lead substantial variations in results, SURE demonstrates consistent performance across different forget set sizes, and when averaging results over multiple epochs, ensuring reliability in real-world scenarios.
Improved Stability in Unlearning:
Reduced Vulnerability to Representation Space Attacks: SURE effectively mitigates vulnerabilities to MIAs targeting the representation space, weakness exhibited by some existing methods. This is achieved through the proposed representation erasure technique and is visualized using t-SNE plots, showcasing the distinct advantage of SURE in enhanc- ing privacy.
PRELIMINARIES
2.1 DANN: DOMAIN- ADVERSARIAL NEURAL NETWORK
We start by describing the key framework that adopt, DANN, introduced by Ganin al. (2016). The authors proposed a simple yet powerful idea for unsupervised domain adaptation with the goal of improved performance under domain shifts. The main insight incorporated into DANN the neural network to learn features (representations) that are: (I) discriminative for the task; and (2) domain-invariant, meaning that the features should not reveal whether the input data came from the source domain or the target domain.
force
Figure provides a schematic overview of the DANN framework. DANN comprises three key com- ponents, each with their own parameterizations:
Feature extractor module Gy(:0 In this work, the backbone of a ResNet-18 18 is used for this module.
Label classification module C-(:;,)); This module takes feature extractor outputs, and maps them to labels. Here, this is a simple fully-connected classification layer with K neurons, where K is the number of classes.
Domain regressor module Ga(:; 0a): This is binary classification component modelling the probability that a given input is from the source vs target domain. In this work we use two-layer fully connected neural networks for this module.
Take feature extractor module trained on some source data, parameterized by fj. Informally, the training process for DANN simultaneously optimizes the following: domain regressor module parameters 02 are trained to distinguish between the source and target features (discrimination task); the feature extractor 0 is fine-tuned to fool the domain regressor; the label classification module 8, is trained to maximize performance feature extractor, a Gradient Reversal Layer (GRL) is inserted between the feature extractor and the domain regressor module. GRL reverses the sign of the gradients flowing from the domain regressor
the source data (classification task). To fine-tune the
2

---

Published as a conference paper at ICLR 2025
Legs
Go,(::0)
#relain
Go. (.00))a))
forgets Pralid
domain
Figure 1: A schematic representation of SURE. The feature extractor is illustrated in blue, classifica- tion module in yellow, and domain regressor in red, with the gradient reversal layer in between the feature extractor and domain regressor. Note that similar schematic representation also captures DANN but with different input distributions.
back to the feature extractor while, during forward propagation, the GRL functions as an identity transformation. This forces the feature extractor to learn features that make difficult for the domain regressor to distinguish between the source and target domains.
2.2
UNLEARNING SETTING
- 1.....,,-11 training dataset, D (Ii, 3)) comprises N i..d. samples drawn from the data distribution. During unlearning, D is partitioned into two mutually exclusive subsets: a forget set Dr, containing samples for which the model must "unlearn" information; and retain set D. = D Dt.
Let X denote the input space, and Y = -o,...,,K K
the set of K possible labels. The source
In addition, we assume access to a validation set Dval- For simplicity of presentation, this validation set is assumed to be of the same size the forget set. Samples in the validation set were not during the original training, may be unlabeled, and are drawn from a target distribution of interest (which could be the same the source distribution). The set to which each sample x belongs is discerned through the domain labels E {0, 1} (the class label here is irrelevant or may not be given). This label is
one for the forget and zero for the validation set.
We use D and DF to denote only the input space samples without the labels from the validation and forget sets respectively; Drest refers the test
UNLEARNING DEFINITIONS AND ASSOCIATED METRICS
3.1 FROM CLASSICAL NOTIONS TO REPRESENTATION-FOCUSST UNLEARNING DEFINITION
A perfect unlearning algorithm would ensure a model behaves exactly as the removed data never existed. This notion can be formalized by drawing parallels differential privacy (Dwork 2006). We begin by defining a measure of closeness between distributions: two distributions be (e, 6)-close if u(B)= S vBB( 8 and v(B) S e' p( for all measurable events B. With this notion of closeness, we can formally define an unlearning algorithm:
are said
Definition 3.1. An unlearning algorithm U is an (c, -unulearre for a learning algorithm A, and training set D if. for all subsets Dj C D of fixed size, the distribution of A(D) De) (A( De) are (e, -ccloe.
This definition captures the essence of unlearning by quantifying the difference between model trained without the forget set and model that underwent unlearning the forget set. However, this raises crucial question: what output distribution
measuring this difference?
In Definition 3.1 the (potentially randomized) learning algorithm .A takes a training set and pro- duces distribution over outputs. A common approach is consider A(.) as returning distribution over loss values. 0 represents the learned weights (which are random variables due to the stochas- tic nature of training), and ((;:0) is the neural network parameterized by then the distribution
3

---

Published as a conference paper at ICLR 2025
considered in previous work is the average loss over the forget set: L(D f ()) Alternatively, one could consider the individual losses of each input in Dr. This approach to evaluating unlearning quality was used in the NeurIPS' 23 competition (Trianaffilluu et al. 2024) and in defining the LiRA membership inference attack (Carlini et al. 2022) which was adapted to the context of unlearning in Hayes et al. (2024).
Another possibility is to directly compare the distributions of the learned weights (0's). However, the high dimensionality and permutation symmetries of neural networks make this comparison chal- lenging, and it remains largely unexplored. A third option involves comparing lower-dimensional representations learned by the network, such as penultimate layer activations, which are usually used for the so-called conjugate kernel computation (c.g., (Hu & Huang] (021)).
Due to the data processing inequality, we know that the information signal, and thus the closeness between A(D \ D:) and (ADD)) Dr) (as measured by (c, cannot be increased when moving from comparing 0's to comparing fDDf :)). Therefore, comparing unlearned models to retrained models based on losses might provide more optimistic view of unlearning performance relative to comparisons based on weights.
f(Df:
In our work, we go beyond comparing losses and evaluate the closeness between A(D \ Dr) and (A));; Dr) using various outputs, including learned weights and internal representations, as de- scribed in Section 3.2
3.2 UNLEARNING METRICS
Efficiency: The primary goal of approximate unlearning is to efficiently and effectively remove the influence of forget samples while achieving performance comparable exact unlearning. To measure computation efficiency, we report the Relative Run Time (RRT), calculated as the ratio of the time taken unlearn the samples the time required for an identical model to retrain from scratch on D, (oracle). To assess effectiveness, we compare the unlearned model to the oracle using various metrics determine how closely it mimics the oracle's behavior. Specifically, we use the following metries:
We compare the unlearned model with the oracle by evaluating the an accuracy gap, i.e., difference in accuracy between the two models, on different subsets of data, each measuring distinct aspects of unlearning (Liu et al.] 2024). Remaining Accuracy (RA), defined as retain set accuracy gap, assesses the fidelity of the unlearning method; Utility, defined as test set accuracy gap, reflects generalization ability; and Unlearning Accuracy (UA), defined as forget set accuracy gap, measures the effectiveness of unlearning. Additionally, for class unlearning scenarios, Class Test Accuracy (CTA) measures test set accuracy gap for the specific class undergoing partial or complete unlearning.
Accuracy Gap:
Canary Unlearning:
To further evaluate the effectiveness of our unlearning method, we consider the concept of unintended memorization. Carlini et al. (2019) observed that neural networks may inadvertently memorize out-of- -isdiitribttionn (OOD) training samples, even though these samples, by definition, unrelated to the target distribution and do not contribute to improving model accuracy. This phenomenon can inadvertently reveal the presence of specific samples in the training data. Specifically, we replace 50 samples from the forget set with OOD samples, referred to as "canaries", as described in (Carlini et al. 22019), creating mixed forget set comprising both in-distribution (IN) and OOD samples. The original model (before unlearning) is trained using this modified training set, which serves as the starting point for all unlearning methods. This experimental setup allows us investigate the robustness our unlearning method in handling unintended memorization and effectively removing the influence of these OOD samples.
Membership Inference Attack: Membership Inference Attacks (MIAs) serve as crucial tool for evaluating the effectiveness of unlearning techniques by assessing the extent to which model has truly "forgotten" the data it was instructed to unlearn. In the unlearning literature, various versions of MIAs have been introduced (Carlini et al.) 2022 Golatkar et al.) 2021 Goel et al., (022b). In this work we use two of the commonly employed ones. The first version (MIA-D) involves training a binary classifier to distinguish between samples from the forget set, Dr, and those from the test set, D Duest: (Kurmanji et al. 2024). This classifier relies on the loss values produced by the unlearned
model when presented with these samples. A higher accuracy of this classifier indicates that the
4

---

Published as a conference paper at ICLR 2025
30
fortor cravp mergi saponge
aoad
coeeh
and
Coeue
spping
R.
aod nort preid mr
Figure 2: Canary outlier experiment results. The forget set, Dr consists of 1000 samples, 50 of which are completely scrambled (OOD) samples), while the remaining samples are randomly se- lected from the training set (IN Samotes hars reflect narrower accuracy gaps, suggesting that the performance is closer to that of the oracle. The summary plot (far right) displays all the plots from the first four panels stacked on top of each other.
unlearned model still retains information about the forget set, making it susceptible to membership inference. The second version of MIA (MIA-II) trains a classifier to differentiate between training, D, and test samples, Drest, based on the model's confidence in its predictions (Liu et al.) 2024 Jia et al. 2023 Song & Mittal (021). This classifier is then used to classify forget samples, Dr, during inference. A higher percentage of forget samples classified as test samples suggests that the model has successfully unlearned the forget set, as its confidence values resemble those of samples it has never seen before. For both types of MIAs, the performance of the unlearned model is compared to that of the oracle model (a model retrained from seratch without the forget set) to assess the effectiveness of the unlearning technique.
Membership Inference Attacks (MIA) are traditionally conducted black-box attacks, where the attacker relies solely on the model's outputs to determine whether a specific data point was included in the training set. In this work, we extend MIA to white-box attacks (Sablayrolles et al. 2019). assuming the attacker has access to the model's 's internal representation space. potentially through its weights. This achieved by implementing modified versions of the two diseussed MIAs. Specif- ically, the modified version of MIA-I involves training a k-Nearest Neighbors (KNN) classifier for a binary classification task, aimed at distinguishing between the forgotten set, Dj, and the test set, Dreat. based on their embedding representations. Conversely, the modified version of MIA-II trains a KNN classifier to distinguish between train samples, D, and test samples, Dtest, based on their representations, subsequently applying this trained KNN to classify the forget samples, D;. While traditional black-box attacks often use logistic regression, we opt for KNN in the embedding space. This choice is motivated by the higher dimensionality of the embedding space compared the out- put space, as well as the potentially small size of the forget set, making a non-parametric classifier more suitable. For brevity, we refer to MIA in Representation Space as MIARS.
To assess the post-unlearning representation in the embedding space, one effective method is to compare the weights and/or activations of the penultimate layer to those of the oracle. However, directly comparing these values is complicated by the permutation invariance property (Ainsworth et al., (022) of neural networks, which arises from the permutation symme- tries of the neurons in each layer (Brea et al. 2019). As noted by Nielsen, multiple arrangements of weights can represent the same underlying function, making it challenging to draw meaningful comparisons between the weights of two separately trained networks (Hecht-Nielsen) 1990p. To ad- dress this issue, we first align the weights of the unlearned model and the oracle using Git Re- Basin, matching algorithm (Ainsworth et al. 2022) designed to find set of permutations that minimizes the distance between the two sets of weights/activationn. Weights are then compared using Euclidean distance.
Weight Space Comparisons:
4 SURE: SELECTIVE UNLEARNING VIA REPRESENTATION ERASURE
Most networks designed for classification tasks can be viewed as consisting of two primary modules: the feature-extractor (G)(=0,)) and the classifier (Gy( 02 Our proposed unlearning-enabled network, SURE, can be viewed as an extension of the original architecture, augmented with an additional domain regressor module during the unlearning process. Unlike in DANN, however, in SURE the domain regressor will be aiming to distinguish between the forget set, and a validation set that the original model never trained on (instead of the source and target domains). In our
5

---

Published as a conference paper at ICLR 2025
NogCrade
Negiooode
Figure 3: Results for random, partial class, and full class unlearning. For each benchmark, the plot displays the absolute value of the gap between the unlearned model and the oracle, so the ideal is to be as close to the center possible. The forget set size is 1000 samples for random unlearning and 100 for partial unlearning. Results are averaged over 5 runs, each with a distinct forget set. To enhance the visual clarity of the plot, normalized each metric's values to the maximum value.
experimental framework, the validation set comes from the same distribution the forget set. The primary goal of the domain regressor module is to assist the network in unlearning in the feature space. This is accomplished by encouraging the network change learned features way that would be discriminative for the primary label classification task, while being non-discriminative when identifying the forget set samples from the validation ones.
Using a similar training procedure as DANN, SURE fine-tunes the original network features to re- move any signature the forget set examples on the learned representation space while maintaining classification performance.
Let T.y and La denote the loss functions for training the label classification module and the domain regressor respectively, defined as
1
La(d, d) = dlog J + (1 d) log
1 d-d' and
Ly( = Σ uh log UN
for arbitrary inputs (d, d) and (g. 3). The network is then trained by performing back poeckpropaggtion on the sum of the following two objectives capturing the classification and discrimination tasks:
(OBJI)
1
Σ
L,(G,(GG((:;))))..)...... and
(3)
DDl
(2.0)ED,
A
(OBJ2)
Σ
La(Ga(R(Cy(1:)))) 1(x E DF)),
(4)
D
REDFUD
where R. represents so-called gradient reversal layer, such that R(xx) on the forward pass and d'(x))/dr the negative identity matrix on the backward pass. In Appendix C we describe how this approach defines saddle-point objective that the domain regressor maximizes and the feature extractor and classification modules minimize. (Note that we assume [Dj = D If not, the sum in Eq. would need weigh each term in inverse proportion the respective size of each set.)
The eettureeexttactor and classification modules, as defined in Section 2.1 are initialized with weights learned during pre-training on D. The domain-regressor initialized randomly.
EXPERIMENTS
In this section, we present our findings showing that our method not only achieves a superior trade- off between performance and unlearning efficiency compared to existing approaches but also ad- method mitigates
dresses some of the limitations of current unlearning techniques. Specifically, the instability issues seen in previous methods, where results can change significantly with minor adjustments to the unlearning setup-compared the used for hyperparameter tuning (Fan al.

---

Published as a conference paper at ICLR 2025
Table 1: Gap in Top-1 and Top-5 accuracy across all accuracy metrics for the four top performing unlearning methods. For each metric, the best values are highlighted in bold.
Top-1 Accuracy
Top-5 Accuracy
Metric
SURE
ISparse
SCRUB
RL
SURE
L.Sparee
SCRUB
RI.
ARA
46.1.10
.03.1.02
1,50.L
7.82.12.09
.42...07
2.15
1.33.1.11
6.37 12.24
AUA
L.+.663
3.0+1.10
2.+2.53
4.2
2.5
3.0=1.46
3.88+2.57
14.64+2..50
Autility
341.27
.624.26
1.264.30
.04.21
.662.16
1.484 .14
1.024
.88= 881
ACTA
844.28
1.04
1.68 1.15
1.34
2.41
2.53
2.24::1.23
3.34
2023). In addition, SURE reduces vulnerabilities to basic membership inference attacks in the rep- resentation space that are exhibited by some other methods. We assess method across three distinct unlearning scenarios: random unlearning. where forget samples are randomly selected from the entire training set; partial class unlearning, where the forget (Dş) is randomly chosen sub- set of a specific class; and class unlearning. where the forget set (Dş) comprises all samples from a particular class, with the primary focus on the partial class unlearning scenario.
116
INNN
CTA
Sumren
12.5
Summary
Figure 4: Gap in Top-1 accuracy for forget set sizes of 500 (top), 100 (middle), and 10 (bottom). All three experiments employed the same set of hyperparameters, tuned specifically for the forget set size of 500. A shorter bar (smaller gap) indicates performance is closer to that the oracle.
5.1 EXPERIMENT SETUP
To assess the stability of unlearning methods, we conduct two sets of exper- iments. First, we report the average accuracy over the top 5 epochs rather than just the best epoch, to ensure that the results are not merely due to anomalously good epoch. Second, we evaluate the methods across various forget set sizes.
Stability Experiments:
We compare our method with several baselines: Oracle, which retrains the model from scratch using only D,: FineTune, which fine-tunes the original model on D,; NegGrad, which uses gradient ascent
Baselines:
Dr: SCRUB (Kurmanji al. 2024). student-teacher framework where unlearning is achieved by minimizing KL divergence between the student and teacher for D, and maximizing it for Dr; BTeacher (Chundawat et al.) 2023). another student-teacher framework with a pre-traine treitriined good teacher and a randomly initialized bad teacher, where the unlearning loss is the sum of KI. divergence between the student and the good teacher for Dr, and between the student and
7

---

Published as a conference paper at ICLR 2025
Nep
Average Emt OND
40-
65+
LieTne
10
10.1
10?
Weight Distance
(a) Weight Distance (I.2) between oracle and unlearned model vs Av- erage Error Gap for each unlearning method.
(c) MIARS-II 1I accuracy gap. At- tacker is trained to distinguish be- tacker trained to distinguish
(b) MIARS-I accuracy gap. At-
tween forget and test examples.
tween training and test examples.
Figure 5: Comparison of the post-unlearning embedding space with that of the oracle using weight comparison (left), MIARS-I attack (middle) and MIARS-II attack (right). The results plotted across all unlearning methods for the partial class unlearning case with forget size of 10 samples.
the bad teacher for Dj; BE CChen et al. 2023) introduces new class for all forget samples, which then removed; BS CChn al. 2023) replaces each forget sample's label with that of similar image from a different class, based on gradient space similarity; LiSparse (Liu et al. (022)) applies L.1 regularization to the objective function to promote sparsity and facilitate unlearning and finally RL replaces forget sample labels with random ones.
is important to note that we faced difficulties achieving good results with NegGrad, BE, and BS due to instability during unlearning. To address this, inspired by Kurmanji et al. (2024), we enhanced these methods by incorporating an additional term into the objective function: cross-entropy loss trained on the D.. We refer to these enhanced methods as NegGrad+, BE+, and BS+ respectively.
Implementation Details: Extensive hyperparameter tuning was conducted independently for each method. The primary hyperparameters optimized were the learning rate and a, where controls the weight of the retain set loss in methods with a weighted loss function that combines losses for the retain and forget sets. Additional hyperparameters, including batch size for the forget set and learning rate scheduling, were also tuned. Optimal hyperparameters for each model were determined through Bayesian optimization achieve the best unlearning utility trade-off. The same original model was used across all unlearning methods, trained for 150 epochs with a learning rate 0.01 and weight decay of 0.0005, without data augmentation, with the learning rate reduced by an order of magnitude at epochs 80 and 150. For the random unlearning method, hyperparameter tuning was carried out with a forget set comprising 5,000 samples. In contrast, for (partial) class unlearning, the forget set was considered to contain 500 samples. In the (partial) class unlearning experiments, we initially selected one class at random (class 5) and conducted all experiments for that particular class.
A Additional Experiments: gain further insight into our method's effectiveness, we conduct ad- ditional experiments involving different datasets, architectures. benchmarks, and analyses, which are not included here due to space constraints. These results are provided in the appendices. More specifically, evaluate method two additional datasets: CINIC- 10 (Darlow al. (018)). an extension of CIFAR-10 augmented with samples from ImageNet, and Tiny ImageNet (Le & Yang 2013) in Appendix D We also test method on ViT-Small, an example of larger model architec- ture, trained on CIFAR-10, as detailed in Appendix D Additionally, we investigate the effectiveness of our method on a non-privacy-related benchmark, Removing Confusion Kurmanji et al. | (2024), in Appendix E, Finally, we conduct sensitivity analysis to evaluate the method's 's robustness to slight changes in hyperparameters, as presented in Appendix
5.2 EXPERIMENT RESULTS
In this section, we examine the effectiveness of our method against both oracle and multiple base- lines across various unlearning benchmarks.
8

---

Published as a conference paper at ICLR 2025
SURE
L15parse
SCRUB
oracle
BTeacher
original
NegGrad4
Figure 6: t-SNE visualization of the post-unlearning embedding space for the entire CIFAR-10 10 training set across all unlearned models. The first column compares these embeddings against the original (top) and oracle models (bottom). Cyan represents retain samples from the forget class, blue indicates other retain samples, and red denotes the 10 forget samples from the same class.
Result reporting: For each experiment, results presented for the best epoch, defined as the one for which the utility versus privacy trade-off is optimized. Since most baselines, except for SCRUB, did not provide an explicit selection process, for reporting results. For SCRUB, the best epoch is selected according to the criteria outlined in the original paper (corresponding the SCRUB-R version of the algorithm). Additionally, unless stated otherwise, the figures and tables in this section do not present the performance of the unlearned mod- els, but rather the gap between their performance and that of the oracle for particular benchmark. The gap is calculated as the absolute value of their differences; thus, smaller value indicates better performance of the method. The difference is averaged over five separate runs, with the original model remaining constant while the randomly selected forget set varies. Error bars represent the variance across these five runs.
applied our own method determine the epoch
Accuracy and MIA: Fig. 3 presents a summary of the results for three key sets of benchmarks: Accuracy metrics, MIAS and RRT, across all three unlearning scenarios and methods (see Ap- pendix B for more detailed plots). Overall, our method outperforms all other unlearning methods in most benchmarks across these scenarios. In the random unlearning scenario, the LISparse method closely follows ours. While SURE outperforms L1Sparse overall all but one of the benchmarks, L1Sparse achieves better RRT. This discrepancy is primarily due to the additional module in our approach-the domain regressor- which tends slow down performance compared to the others. Regarding MIA attacks, our results indicate that these attacks are largely ineffective in this scenario, with all methods, including the oracle, yielding similar outcomes. In contrast, we observed that these attacks are more effective in class unlcarning scenarios. Notably, our method excels in these scenarios, outperforming all other methods across all accuracy benchmarks and both MIA attacks while maintaining competitive RRT.
Canary Unlearning: Fig. presents the accuracy results from the canary unlearning experiment. SURE outperforms all other methods across all accuracy measures. It effectively unlearns both the IN and OOD samples while maintaining strong generalizability (Utility) and fidelity (RA).
The primary motivation for our method is the hypothesis that a cor- relation exists between the similarity of the post-unlearning model's representation to that of the oracle and the effectiveness of the unlearning process. To investigate this hypothesis, we plot the Average Error Gap against the L2 distance between the penultimate layer's weights of the unlearned model and the oracle, as shown in Fig. Sa| The Average Error Gap is defined the average abso- lute value of the difference between the unlearned model and the oracle across five key unlearning benchmarks: UA, Utility, RA, CTA, and MIARS-I. The results clearly indicate correlation be-
Representations and MIARS:
9

---

Published as a conference paper at ICLR 2025
tween the alignment of the unlearned representation with the oracle and the success of unlearning. In addition, our method has the lowest Average Error Gap and second smallest weight distance, After LiSparse. The t-SNE visualization of the embedding space for all unlearned models, shown in Fig. [ further suggests a correlation. Notably, the most effective unlearning methods, SURE followed by L.Spaare, exhibit representations that closely resemble the oracle.
In addition to subpar unlearning performance, discrepancies between the unlearned model and the oracle in the representation space can render the model vulnerable simple MIAs. The results are shown in Fig. [5 and [cc As with previous plots, the y-axis indicates the difference in MIA accuracy between the unlearned model and the oracle. The model unlearned with SURE exhibit is most successful in defending both attacks. Also, overall models whose representation is closer to oracle (measured by weight distance) are less susceptible to attacks.
Stability: For an unlearning method to be effective in real-world applications, it must demonstrate stability. We assess stability by conducting two sets of experiments. In the first set of experiments, report the top-5 accuracy, calculated the average accuracy over the top 5 epochs for the four highhes-performing unlearning methods, in Table T We compare these results to the top-1 accuracy, which reflects the performance from the best epoch. Our findings indicate that SURE outperforms the other three methods overall and across most individual metrics.
In the second experiment, perform unlearning for different forget sizes (100 and 10), other than the one used for hyperparameter tuning (500). This experiment assesses how sensitive each unlearn- ing method is slight changes in the setup for which it is optimized. The results are presented in Fig. 4 While L.Sparse and SCRUB both show performance comparable to SURE for the forget set size used for hyperparameter tuning (top plot), the gap between their performance and that of SURE widens as the difference between the forget set used for unlearning and the original forget set grows (middle and bottom plots).
CONCLUSION AND FUTURE WORK
In this work, we propose SURE, an approximate unlearning method that focuses unlearning in the representation space rather than solely in the output space. This approach is motivated by observation that achieving similarity with the oracle in the representation space leads more successful unlearning. Through extensive experiments, we demonstrate that our method consistently yields superior or comparable results compared to existing techniques across a range of scenarios and for a variety of unlearning benchmarks, including multiple accuracy metrics and two MIAs. Furthermore, we empirically validate that the post-unlearning representation of the unlearned model closely resemble that of the oracle. Finally, our approach exhibits greater stability and reduced susceptibility to white-box attacks the representation space, enhancing overall model security.
One observed limitation of our method is the need for more unlearning updates compared to other techniques, especially the random unlearning scenario where forget samples are randomly se- lected from any class. A potential solution could be to employ class-conditional discriminator in- stead of a generic one, allowing the discriminator to focus on differentiating between class-specific forget and validation samples. While promising, such an approach would come cost of training and storing multiple discriminators.
ACKNOWLEDGEMMET
We thank Mila - Quebee AI Institute and Google DeepMind for providing the computational re- sources that supported this work. We also sincerely appreciate Devin Kwok for his valuable input and for providing the implementation code, both of which were instrumental for our analyses in the representation space. We also thank Katja Filippova for providing feedback on the drafts of this work. Daniel M. Roy is supported by the funding through NSERC Discovery Grant and Canada CIFAR AI Chair at the Vector Institute.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
To ensure reproducibility, we provide a comprehensive outline our experimental setup in Sec- tion 5.1 including details on hyperparameters, hyperparameter tuning strategies, datasets, and al- gorithms. The precise hyperparameter settings are further documented in the Appendix [ Our stability experiments, as detailed in Section [.2] significantly enhance reproducibility and reveal herent instability in some previously proposed methods. Furthermore, we address critical and often overlooked aspect of unlearning algorithms Section 5.2 by carefully outlining our stopping crite- ria selection process. Finally, all average numerical values in our experimental results are presented with error bars computed over 5 independent runs.
REFERENCES
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209. 04836, 2022.
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss land- scape. arXiv preprint (xXiv:/90..029/1 2019.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium security and privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pp. 267-284, 2019.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Mem- bership inference attacks from first principles. In 2022 IFFE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the TEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7766-7775, 2023.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7210-7217, 2023.
R Dennis Cook and Sanford Weisberg, Criticism and influence analysis in regression. Sociological methodology, 13:313-361, 1982.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. Cinic- is not imagenet or cifar-10. arXiv preprint arXiv: 1810.03500 2018.
Cynthia Dwork. Differential privacy. In International colloquium automata, languages, and programming, pp. 1-12. Springer, 2006.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-basee weight saliency in both image classification and generation. arXiv preprint 2r /2508, 2023.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. aodain-adversarial training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
Shashwat Goel, Ameya Prabhu, and Ponnurangam Kumaraguru. Evaluating inexact unlearning requires revisiting forgetting. CoRR abs/2201.06m 2022a.
Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint MoXiv:2201.066-4 2022b.
11

---

Published as a conference paper at ICLR 2025
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacc forgetting in deep networks. Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 792-801, 2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.0300 2019.
Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid false sense of privacy. arXiv preprint arXiv:2403.012/8, 2024.
Robert Hecht-Nielsen.
On the algebraic structure of feedforward network weight spaces. In Ad- vanced Neural Computers, pp. 129-135. Elsevier, 1990.
Zhengmian Hu and Heng Huang. On the random conjugate kernel and neural tangent kernel. In International Conference Machine Learning, pp. 4359-4368. PMLR, 2021.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data dele- tion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016. PMIR, 2021.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsification simplify machine unlearning. arXiv preprint arXiv:2304.0499 04934444 1(2):3, 2023.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)33 2015.
Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, Sijia Liu, et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. In International Confer- ence on Muchine Learning, pp. 5558-5567. PMLR, 2019.
Liwei Song and Prateek Mittal. Systematic evaluation privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pp. 2615-2632, 2021.
Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we mak- ing progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.09073, 2024.
12

---

Published as a conference paper at ICLR 2025
SLIEE
L1Sparse
SCRUB
Bllaec wet
NegGrad4
RE4
R6
Driginal
Utillyy
100
07
30
60-
50
02
40
10
15
15
20
10
Epoch
Spoch
Spoch
Figure 7: Top-1 accuracy for the three accuracy metrics, as indicated in the title of each plot, during unlearning across all unlearned models for the random unlearning case. In contrast to the results section, the values shown here represent the actual accuracy rather than the gap. The gray dashed line in each plot illustrates the performance of the oracle.
A RELATED WORK
Unlearning first introduced in (Cao & Yang 2015) who proposed forgetting algorithm for sta- tistical query learning. Nowadays, a plethora unlearning methods have been proposed that attempt to selectively remove the influence of the forget set from the trained model. [Golatkar al. (2020a) introduced an information-theoret method that uses a Newton step and adds noise to erase the for- get set. Golatkar al. (2020b) proposed NTK forgetting, employing first-order Taylor expansion estimate the weights that would have been obtained if training without the forget set. These methods work well under assumptions of stability of SGD. Izzo et al. (2021); Koh
Koh & Liang (2017) proposed unlearning algorithms based on influence functions Cook & Weisherg (9822). drawing connections to (c, forgerting (Guo et al. 2019). Goel al. (2022)) introduced two approximate unlearning methods: Catastrophic Forgetung K(CF-)
-orgetting-((FF-K) and Exact Unlearning-k (EU-k). In both methods, the first A layers the pre-trained model are frozen. CF-K, the remaining layers are fine-tuned on D, while in EU-k forgetting, they are trained from scratch D,. (Chundawat et al. 2023 Kurmanji eet al. 2024) proposed teacher-student formulations, where the teacher the original model and the student is initialized from the teacher and, through specially-crafte
distillation procedures (where the teacher is frozen and the student is updated), turns into the unlearned model. In (Chundawat et al., 2023), the student distills retain set knowledge from the teacher, while being encouraged to agree with another "incompetent" teacher on the forget set, whereas SCRUB (Chundawat et al.) 2023) considers only one teacher, and encourages the student agree with it on the retain set while disagreeing with it on the forget set. Chen et al. ((Chen et al. 2023) adopted a unique strategy for class unlearning by adjusting the decision boundary of the original model mimic the oracle's decision-making They introduced two boundary shift methods: Boundary Expanding, which cre- ates a new class for all forget samples that will later be removed, and Boundary Shrinking, which assigns forget samples labels from similar images in different classes. Liu et al. (2024) proposed LISparse, adding an Ll-penalty into the fine-tuning baseline, after conducting an investigation that shows that sparsity makes unlearning easier. Fan et al. (2023) proposed localized unlearning ap- proach that targets only a subset of parameters that are deemed-to-be the most critical for the forget set.
B UNLEARNING RESULTS FOR CIFAR-10
Fig. and Fig. I0 illustrate the accuracy metrics (UA, RA, and Utility, and in the case of class unlearning, CTA) during the unlearning process across different methods for random and partial class unlearning scenarios respectively. The gray dashed line in each plot represents the oracle performance.
The label Original denotes the original model which serves as the starting point for all unlearning methods. To evaluate the effectiveness of each unlearning method we compare the performance of the unlearned model with that of the oracle on all accuracy metrics. For each unlearning method, the comparison is made the best epoch. The absolute value of the difference between the unlearned model and the oracle model for each method is displayed in Fig. g and Fig. [T] In the random unlearning scenario, the forget consists of 1,000 samples randomly selected from the entire
13

---

Published as a conference paper at ICLR 2025
UA
Liility
PA
Summary
12.5
53
mostow LiSpore uratie
Oegonejle
Aire noatow Lispeta scace
Figure 8: Gap in Top-1 accuracy for various accuracy metrics, as indicated the title of each plot, for the random unlearning scenario with forget size of 1000, across all unlearned models. Results are reported for the best epoch.
training set, while the partial unlearning scenario, consists of 100 samples randomly chosen from class 5.
0.70
0.50
0.90
03
0.55
0.50
0.4
Wortone LXSaure SCRUB
Diginal
Now
Lxspurne
Orgjna
Figure 9: MIA-I (left) and MIA-II (right) in output space across all unlearned models for random unlearning. Red line represents the performance of the oracle.
The results for MIA-I and MIA-II in the random and partial unlearning cases are shown in Fig. and Fig. [2 respectively. The red line indicates the attack accuracy on the oracle, serving as a reference point for all unlearning methods. MIA-I does not appear to be effective in the random unlearning scenario, as nearly all unlearning methods seem equally successful at defending against it. However, this is not the case for the partial class unlearning scenario, where method's 's success rate is very close to that of the oracle. For MIA-II, our method outperforms the others, showing an overall success rate closest to the oracle's for both random and partial unlearning scenarios, although the results for the random scenario are quite similar across all methods.
L1503100
Prezasc
MILSCTOr
70
Epach
Eooch
Qaach
Figure 10: Top-1 accuracy across all unlearned models for the partial class unlearning case with the forget size of 100. Each panel presents different accuracy metric, as indicated the title. The gray dashed line in each plot illustrates the performance of the oracle.
14

---

Published as a conference paper at ICLR 2025
LA
USBIN
54
Samerrry
repinns
tepingg
Figure 11: Gap in Top-1 accuracy for various accuracy metries for the partial unlearning all unlearned models.
0.75
0.65
344
02
0.50
Hor
Hegtraad
de
Orginn
the
mancher
Origina
Figure 12: MIA-I (left) and MIA-II (right) results for the partial class unlearning scenario. Red line represents the performance of the oracle.
C GRADIENT UPDATES
The purpose of this section is to describe the mathematical optimization problem underlying train- ing. We will see that is implemented succinetly with a gradient reversal layer (GRL), which is how we describe it in the main paper (Eqs. (3) and (4)). Readers may find one of these two equivalent descriptions more intuitive.
Define
E(0,,0g,00 = |Del
1
Σ I,(C,(G(((:))))))))..
(aN)ED,
LICGaGGy(s;0)))))))), 1(x DF).
D
REDFUDE
This objective is the objective underlying domain adversarial neural networks (Ganin et al., 2016). The first term captures the training error on the classification task and depends the feature extrac- tor and classification modules. The second term captures the error on the discrimination task, and depends the feature extractor and domain-regreen reegessor modules.
We optimize this objective for training in SURE. In particular, the goal of domain adversarial training to find saddle point, where the optimal parameters for each module satisfy
(6, = arg min F(O,, 0y, 0a), 9,
(5)
0. arg max E 8y 0z).
(6)
15

---

Published as a conference paper at ICLR 2025
UA
Utility
CTA
MIA
MIA II
ART
SCRUB
SCRLB
fineTuna
Dpprre
Expparse
Rlearjet
SURE
DE-
BE+
SrdSrade
egrrdd+
85+
RL
Figure 13: Results for the random and partial class unlearning scenarios on CINIC-IO. Each bench- mark plot illustrates the absolute difference between the unlearned model and the oracle. For ran- dom unlearning, the forget set contains 1,000 samples, while for partial unlearning, it consists of 100 samples. The results are averaged over five distinct runs, each with unique forget set. To enhance visual clarity, the values for each metric have been normalized to their maximum value.
Following Ganin et al. (2016), use the following gradient updates seek saddle-point:
aLg aLd 00
Of - Of 00,
(7)
aLy
0g + 0 1 00,
(8)
aT.d
0g + 0g μn
00a
where is the learning rate. As discussed in (Ganin et al. 2016), these updates closely resemble SGD updates for simple feed-forward deep model objective function, with one key difference: the gradients from the domain regressor are subtracted instead of added. This adjustment ensures that, while the regressor's task distinguish between forget and validation samples, the feature extrac- tor aims to deceive the regressor. As result, the feature extractor learns set of indiscriminative features that cannot differentiate between the forget samples and previously unseen samples.
L1sparse
SCRU3
SCRUB
UURE
L35padee
URE
Hino HunO
Hrelune
Figure 14: Results for the random and partial class unlearning scenarios on the Tiny ImageNet dataset. For random unlearning, the forget contains 1,000 samples, while for partial class un- learning, it contains 400 samples. Results are averaged over five runs, each with a unique forget set.
As noted by Ganin al. (2016). this modification to the usual update rules is cumbersome to im- plement naively in most deep learning libraries. To address this, the authors introduced a special "gradient reversal layer" (GRL), which has associated parameters. During forward propagation,
16

---

Published as a conference paper at ICLR 2025
SENUS
Lisparie
Pire..n
Figure 15: Results for the random and partial class unlearning scenarios using Small-Vil trained on CIFAR-10. For random unlearning, the forget set contains 1,000 samples, while for partial class unlearning. it contains 10 samples. Results are averaged over five runs, each with unique forget set.
the GRL acts identity transformation. During backpropagation, however, it takes the gradient from the subsequent layer and reverses its sign (i.e., multiplies it by the negative identity) before passing it to the preceding layer. Because the computation involving the domain regressor weights comes after the GRL, gradient descent minimizes the discrimination task objective in the domain regressor weights. In contrast, the computation involving the feature extractor weights come before the GRL, and so gradient descent maximizes the discrimination task objective in the feature extractor weights.
In conclusion, if we perform backpropagation of the computation representing of Eq. and Eq. 4., the latter of which includes the GRL, we obtain updates identical to Eq. 7 This objective with GRL can be optimized using standard auto diftoodiffeeettiiiionn tools.
Method
Retain Err
Forget Err
Test Err
IC Retain
FGT Retain
IC Test
FGT Test
SURE
0.08 + 0.11
12.25
4.57
22.75
0.31
+ 0.0
0.4 0.49
0.16 0.01
29.4 3.88
SCRUB
0.32 0.3
10.5 2.03
25.36
0.0 0.0
4,0 4.86
0.17+
35.6 6.56
L
0.08 + 0.11
12.25 3.1
22.18
0.53
0.0 0.0
0.2 0.4
0.15
27.0 3.63
Tine Tume
2.77 1 0.73
29.75
26.89 1 0.46
0,07 1 0.01
2.6 1.85
0.32
174 3.61
BE+
0.0
100.0
28.61
0.58 0.12
25.0 13.61
0.64
1000
BS+
2.13
8.975
34.14
0.07 + 0.04
7.8 7.8:156 15.6
0.45
13.0
RL
1.25
19.75
26.53
0.07
60.4 14.36
0.01
3844 6.44
NegGrad+
0.0 0.0
100.0+0000
28.6 0.58
0.5 0.0
3960.0 + 0.0
0.5 0.0
1000.0 + 0.0
BTeacher
1.17 + 0.29
8.75 3.45
27.67 +: 0.57
0.05 + 0.01
2.0 + 2.61
0.35 0.01
11.6 + 3.2
Table 2: Performance comparison of multiple unlearning methods on the RC task across various metrics. Results are averaged over 5 runs.
D MORE UNLEARNING RESULTS
In this section, we evaluate the performance of our unlearning method across additional datasets and architectures. Specifically, we extend our analysis to CINIC-10, Tiny ImageNet, and ViT-Small, comparing our method against other unlearning approaches across various benchmarks.
In Fig. 13 we compare the performance of our method on the CINIC-10 dataset against all other baselines across various unlearning metrics, similar to Fig. 3 The figure displays the absolute difference between the unlearned model and the oracle for all accuracy metrics (UA, RA, Utility, and CTA), well as for RRT and MIAs. We observe that SURE outperforms most benchmarks in both scenarios. Consistent with the results from CIFAR- 10, LISparse ranks as the second-best performing method, occasionally surpassing SURE few metrics.
In addition, we evaluate the performance of our method on the Tiny ImageNet dataset, comparing against three other top performing methods identified in previous experiments: L1Sparse, Tune, and SCRUB. The results, presented in Fig. 14] show that our method outperforms all others across all benchmarks except for RRT, consistent with findings CIFAR-10 10 and CINIC-10. For random
17

---

Published as a conference paper at ICLR 2025
400
404
30
LED
LGD
60
040
08+
500
508
300
930
340
6
633
D4O
SCD
APC
3:30
021
8:4
D4T
02+
343
4:55
065
0.5
K15
6
CL
K
4.5
6:44
6:3
0.33
635-
931
2004
4000
311
222
8:30
4:21
6:0
405
500
0:0
6:00
0.0
0.0
308
481
EM
D22
A02
S6C
523
0:00
6:00
030
080
082
032
D42
322
585
35C
2:4
0:30
AM
057
029
090
AU9
Figure 16: Heatmaps illustrating the sensitivity analysis of SURE's performance with respect to the two key hyperparameters: (r and a, across the four main accuracy benchmarks: UA (top-left), Utility (top-right), RA (bottom-left), and VTA (bottom-right). Each heatmap shows performance variations relative to the best-tuned hyperparameter settings.
unlearning, we were unable to identify set of hyperparameters for FineTune and SCRUB that achieved successful unlearning. Both methods fail to achieve the desired unlearning-utilit tradeoff, as they completely unlearn all information. In the case of class unlearning. FineTTune and SCRUB are more competitive; however, the primary competition remains between our method (SURE) and I. L.Spaas,, with SURE outperforming Sp in nearly all benchmarks.
To explore the effectiveness of our method on larger models, we tested its performance using ViT- Small across both random and class unlearning scenarios. Results are presented in Fig. [5] For random unlearning, our method demonstrates a slight edge over LiSparse. We were unable to find hyperparameters for SCRUB that enabled effective unlearning, as models using this method failed to achieve the desired unlearning-utte utility tradeoff. For class unlearning, the results are more balanced: our method outperforms LISparse on out of 7 benchmarks. while LiSparse outperforms on the remaining 3. These findings highlight the robustness of our approach, while also demonstrating that pruning-based methods like Spaare can remain competitive in specific settings.
E UNLEARNING RESULTS FOR THE RESOLVING CONFUSION TASK
In this section, we apply our unlearning method the Resolving Confusion (RC) task introduced in Kurmanji et al. (2024). This task serves as an example of non-privacy-rellted application where unlearning is beneficial for resolving confusion between classes caused by a portion of the model's original training set being mislabeled.
We evaluated the performance of SURE on the RC task using a ResNet-18 model trained on the CIFAR- 10 dataset. We compared our method against all eight competing unlearning methods intro- duced in the paper. In our implementation, the forget set consists of a randomly selected subset (1%) of training samples from classes 0 and 1, which are intentionally mislabeled as the opposite class. Given that all samples in the forget set are mislabeled, the objective of unlearning in this context is to resolve the induced confusion. An unlearning method is considered successful if, the end of the unlearning process, the forget samples are correctly relabeled while maintaining the model's performance on the test and retain sets.
To evaluate the performance of our unlearning method, we report the forget error (Forget Err) along- side the test (Test Err) and retain errors (Retain Err) An effective method is expected to exhibit low
18

---

Published as a conference paper at ICLR 2025
values across all three metrics. In addition to these, and in line with the original paper Kur- manji et al. (2024), we also report the Interclass Confusion Error (IC-ERR) and FGT-ERR. Unlike the previously introduced error,, which count all types of incorrect predictions, IC-ERR specifically measures errors involving the confused classes (in our setup, class and class 1). It counts instances where sample from class is predicted to belong to any other class, and vice versa for class 1. FGT-ERR, on the other hand, is even more specific, as it counts only cases where a sample from class is mislabeled as class or sample from class I is mislabeled as class 0. IC-ERR and FGT-ERR are reported for both retain and test sets. Our results presented in Table demonstrate that SURE the top- pprperfonning approach, alongside LISparse, yielding similar results for many metrics.
F SENSITIVITY ANALYSIS OF HYPERPARAMETERS
To assess the sensitivity of our method's performance to variations in its hyperparameter values, we conducted a sensitivity analysis focusing on the two most critical hyperparameters: a and the learning rate (Ir). This analysis was carried out for the class unlearning scenario using a ResNet- 18 model trained on the CIFAR-10 dataset, with a forget set consisting of 100 samples. Using a grid search approach, we varied each parameter's value within range of -20% to +20% around its optimal value and analyzed the resulting changes across multiple unlearning benchmarks.
The observed performance variations illustrated in Fig- 16 The figure presents four heatmaps, each providing a sensitivity analysis for a specific accuracy benchmark as a function of these hyper- parameters. These heatmaps highlight the differences in performance relative the model trained with the best hyperparameter configuration from tuning. The visualizations reveal the sensitivity to each parameter and their interactions, showing that, compared to other methods (see Section 4 and Figure in Fan et al. (2023)) SURE demonstrates low sensitivity to hyperparameter variations. For comparison, Fan et al. (2023) reports fluctuations of approximately 40 points in the average perfor- mance gap between the unlearned model and the oracle for the Influence Unlearning method. While there is always room for improvement, our results suggest that sensitivity to hyperparameters is not significant limitation of SURE.
G UNLEARNING HYPERPARAMETERS
The optimized values for all hyperparameters across unlearning methods in both random and partial class unlearning scenarios are presented Table and Table respectively. The main hyperparam- eters we tuned in this work are the learning rate, lr, (retain set loss weight in unlearning methods with a weighted objective function), forget set batch size, and the learning rate scheduling, defined by the epoch at which the learning rate drops.
Params
SURE
LISparse
SCRUB
FineTune
Bteach
BE+
BS+
RL
NegGrad+
rl
.045
.0045
0008
.0043
.035
.0092
.0079
.0671
.0072
a
.877
.00044
.997
.954
.982
.998
.991
fbatch size
256
32
256
256
8
128
128
32
Ir decay
10
10
18
8
8
14
10
14
18
Table 3: Optimized hyperparameters for the random unlearning scenario: learning rate (Ir), a, forget set batch size (fbatch size), and the epoch for learning rate decay (Ir decay).
Params
SURE
L1Sparse
SCRUB
FineTTnee
Bteach
BS+
BE+
RL
NegGrad+
Ir
.0372
.0088
.001
.0025
.0320
.0047
.0058
.0014
.0694
a
.4241
.00024
.6089
.3453
.9841
.4639
.9881
fbatch size
128
256
32
64
128
256
Iy decay
4
8
12
Table 4: Optimized hyperparameters for the (partial) class unlearning scenario: learning rate (lr), o, forget set batch size (fbatch size), and the epoch for learning rate decay (Ir decay).
19


Reviews:
[{"review_id": "Sdy7oE7LKj", "reviewer": "Reviewer_ezzj", "summary": "The paper proposes a novel algorithm – SURE, for targeted unlearning. Inspired by work in domain adaptation, SURE treats a “forget set” and a validation set from the same distribution as data from two different domains, and optimizes the feature extractor to be s.t., the representations of the forget data are obfuscated from a discriminating domain classifier. Since SURE works directly on representation space, it prevents the stealing of information that may have remained in the latent space. The authors conduct experiments across a wide variety of utility metrics and compare their performance with many existing works.", "questions": "- I would like to see results on different architectures and possibly more benchmarks. \n- I would like to hear the authors' opinions on the practical implications of assuming access to the validation set.\n- I would like to hear the authors' opinions on why SURE performs particularly (relatively) worse on RRT compared to other metrics.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to understand. \n- SURE demonstrates impressive results over existing works on a wide variety of metrics.\n- SURE provides a stable training regimen for unlearning methods that are usually sensitive to hyper-parameters.", "weaknesses": "- The experiments are limited to CIFAR-10 and ResNet-18, albeit with a few results on CINIC-10. I believe that it’s important to observe the results on a wider variety of benchmarks, and more importantly models (like ViTs, etc) to ensure generality.\n- Access to the validation set from the same distribution as the forget data: I am unsure how practically valid this assumption might be as the kinds of data people may want to remove are also primarily scarcely available, and if you trained and now want to unlearn on all of it (which seems plausible for current foundation models), it would not sit well with your current setup\n- Novelty: The paper acknowledges that it heavily builds up on the DAN framework and applies it to the machine unlearning setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nHXEQsNVEP", "reviewer": "Reviewer_e2Uw", "summary": "This paper proposes an approximate unlearning method that focuses on unlearning in the representation space rather than in the output space. Experiments demonstrate that the method has superior or comparable results compared to existing techniques.", "questions": "1. The paper is mainly focused on classification tasks. I'm wondering if it can be extended to multi-modal tasks (image-text) and generation tasks (LLM, Diffusion)?\n\n2.  Why does the reversal layer R(x)=x can helps to forget the representation & knowledge of D_f? Could the author provide more details about the optimization goal (2) on Page 6? (I see the optimization details in Appendix C, but still confused.  What is the relationship between L_d(G_d(R(G_f(x;\\theta_f));\\theta_d), 1(x\\in D_f) ) and equation (7) (8) (9) in Page 14 & Appendix C?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel machine unlearning method for classification tasks (CIFAR-10, CINIC-10) via domain adversarial training, which shifts the focus from the output space to the representation space.\n\n2. The performance gain from the proposed method is convincing.\n\n3. The presentation is clear and easy to follow.", "weaknesses": "1. The paper compared its performance with 'Towards Unbounded Machine Unlearning, NIPS 2023'; ' by the MIA metric (User Privacy Settings). However, the proposed method should also be evaluated in removing biases (RB), Resolving Confusion (RC) settings. RB aims to achieve the highest possible error on the forgetting set, without hurting the error of the retain and the test set. It evaluates the performance of the proposed algorithm in unlearning the knowledge of the target domain while not hurting other's performance.  RC aims to resolve confusion between two classes that the original model suffers from due to a part of its training set being mislabelled. (settings from ''Towards Unbounded Machine Unlearning, NIPS 2023''）\n\n\n2. Although the authors use CINIC-10 (a subset of ImageNet) to validate the effect of forgetting, the authors should still provide the effect of forgetting on the ImageNet-1K. There are many pre-trained models on ImageNet-1K and this study focuses on classification tasks, it is more convincing to test on ImageNet-1K to prove the real-world applicability of the proposed algorithm.\n\n\n3. The title is ambiguous and should emphasize domain adversarial training rather than adversarial training to make it more clear, as domain adversarial training aims to improve the domain adaptation ability of the target model via an adversarial fashion, while adversarial training aims to improve the target model's robustness against adversarial attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OXlhNkRP1u", "reviewer": "Reviewer_5UH3", "summary": "The paper proposes SURE (Selective Unlearning via Representation Erasure), a novel approach for targeted unlearning in neural networks, based on representation erasure rather than model output manipulation. Using concepts from Domain-Adversarial Training of Neural Networks (DANN), SURE aims to align the representation space of a model post-unlearning with that of an oracle model. This method focuses on defending against more robust attacks, particularly membership inference attacks (MIAs) in the representation space, improving the stability of unlearning while preserving model utility.", "questions": "1. Could the authors provide a breakdown of the computational cost of SURE compared to the baselines?\n2. How sensitive is SURE’s performance to the choice of hyperparameters, especially those determined by Bayesian optimization? If SURE requires highly tuned hyperparameters, exploring ways to reduce this dependency could make it more accessible for practical applications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a unique perspective by targeting the representation space instead of the output space, addressing a critical gap where unlearned models might retain information in their intermediate representations. This angle has the potential to advance understanding and methodology in the field of machine unlearning.\n- The authors validate SURE’s effectiveness using extensive experiments across several unlearning scenarios, including random, partial, and full class unlearning. The results consistently demonstrate that SURE achieves a favorable trade-off between performance and unlearning efficiency.\n- By specifically aiming to improve defenses against membership inference attacks within the representation space, SURE contributes to increasing the security and robustness of unlearned models.\n- The technical approach, particularly the adaptation of DANN with gradient reversal and domain-adversarial training, is well-articulated.", "weaknesses": "- Although SURE demonstrates stability across different forget set sizes, its reliance on careful hyperparameter tuning (with Bayesian optimization) may hinder ease of implementation for practitioners who cannot afford extensive hyperparameter searches.\n- SURE appears to introduce additional computational complexity. While this approach enhances unlearning quality, it may limit SURE's scalability, especially for larger models or datasets where computational efficiency is crucial.\n- Minor Comment on Notation Consistency: The authors should maintain consistent notations across the paper to improve readability. For example, in Figure 1, the notation for different modules should be consistent with those used in the main text ($G_f(., \\theta_f)$, $G_y(., \\theta_y)$, and $G_d(., \\theta_d)$). This would help readers follow the mathematical formulations and understand the relationships among components more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)