Adversarial Machine Unlearning
Authors: Zonglin Di
Sixie Yu
Yevgeniy Vorobeychik
Yang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Machine unlearning
Adversarial approach
Stackelberg game
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: A Stackelberg game-based machine unlearning method

Abstract: This paper focuses on the challenge of machine unlearning, aiming to remove the influence of specific training data on machine learning models. Traditionally, the development of unlearning algorithms runs parallel with that of membership inference attacks (MIA), a type of privacy threat to determine whether a data instance was used for training. However, the two strands are intimately connected: one can view machine unlearning through the lens of MIA success with respect to removed data. Recognizing this connection, we propose a game-theoretic framework that integrates MIAs into the design of unlearning algorithms. Specifically, we model the unlearning problem as a Stackelberg game in which an unlearner strives to unlearn specific training data from a model, while an auditor employs MIAs to detect the traces of the ostensibly removed data. Adopting this adversarial perspective allows the utilization of new attack advancements,  facilitating the design of unlearning algorithms. Our framework stands out in two ways. First, it takes an adversarial approach and proactively incorporates the attacks into the design of unlearning algorithms. Secondly, it uses implicit differentiation to obtain the gradients that limit the attacker's success, thus benefiting the process of unlearning. We present empirical results to demonstrate the effectiveness of the proposed approach for machine unlearning.

Published as a conference paper at ICLR 2025
ADVERSARIAL MACHINE UNLEARNING
Zonglin Di Sixie Yu²*, Yevgenly Vorobeychik', Yang Liu 'f
University of California, Santa Cruz, Stellar Cyber, Inc., "Washington University St. Louis 1 (zdi, yangliu) wwcsc. Pbit. www.susssiiieee g com Syrorobecchkkeusst...e
ABSTRACT
This paper focuses on the challenge of machine unlearning, aiming to remove the influence of specific training data on machine learning models. Traditionally, the development of unlearning algorithms runs parallel with that of membership inference attacks (MIA), a type of privacy threat to determine whether data instance was used for training. However, the two strands are intimately connected: one can view machine unlearning through the lens of MIA success with respect to removed data. Recognizing this connection, we propose game-theoretic frame- work that integrates MIAs into the design unlcarning algorithms. Specifically, we model the unlearning problem as a Stackelberg game in which unlearner strives to unlearn specific training data from a model, while an auditor employs MIAS to detect the traces of the ostensibly removed data. Adopting this adver- sarial perspective allows the utilization of new attack advancements, facilitating the design of unlearning algorithms. Our framework stands out in two ways. First, it takes an adversarial approach and proactively incorporates the attacks into the design of unlearning algorithmss. Secondly, it uses implicit differentia- tion to obtain the gradients that limit the attacker's success, thus benefiting the process of unlearning. We present empirical results to demonstrate the effective- ness of the proposed approach for machine unlearning. The code is available at com UCCCC SEECC-EELL/SSS SG-Unneannnnn
INTRODUCTION
The enactment of the General Data Protection Regulation (GDPR) by the BU has elevated the importance of deleting user data from machine learning models to a critical level. This process is distinctly more intricate compared removing data from conventional databases. Erasing the data's 's imprint from machine learning model necessitates approach to negate the data's 's influence on the model comprehensively while maintaining the utility and accuracy of the model.
Beyond this, establishing the true extent to which data influence has been erased from the model poses significant challenge (Song & Mittal 202T). Numerous methods and metrics have been advanced to validate the thoroughness of data removal, each with varying degrees of reliability and efficacy (Guo) ctt al. 2020 Thudi et al.] 2022b). We propose a novel adversarial perspective on unlearning that we argue is more robust framework for effective machine unlearning. In this approach, the focus shifts to simulating possible attacks aimed at inferring whether the data that should have been forgotten nevertheless maintains some influence on the model. If, within this adversarial framework, attacker fails to distinguish whether a data point was part of the training set or merely a typical instance of unseen data, we can conclude that the influence of the data point on the data has been successfully unlearned.
We leverage advancements from the burgeoning domain of Membership Inference Attacks (MIA) to simulate an adversary (Shokri et al., 2017), therein framing Stackelberg Game (SG) between an unlearner, tasked with orchestrating the unlearning process, and an auditor deploying MIA to deduce the membership of data in the model's training set. The key idea is for the unlearner to adjust the model being unlearned by utilizing gradient feedbacks from the auditor's 's optimization
Equal Contribution. *Corresponding author: Yang Liu <yangl waangglv@ucsee uu@uccc.

---

Published as a conference paper at ICLR 2025
problem, moving the model in a direction that limits the effectiveness of the attack, thus achieving the goal of unlearning. Specifically, we formulate the MIA as a utility- maximizig problem, where the utility measures the remaining influence of data point in the unlearned model. The unlearner's loss function is defined as a combination of the degradation of model performance and the auditor's utility. We harness the development from implicit differentiation and design a gradient-based algorithm to solve the game, allowing for seamless integration into existing end-to-end pipelines (Gould et al. 2016
Amos & Kolter 2017 Agrawal et al. 2019b).
The contributions of the present paper are summarized below
We propose to evaluate the effectiveness of an unlcarning algorithm from an adversarial perspective, inspiring us to develop game theory framework that enables the of advanced MIAs for enhancing the unlearning process.
Additionally, we design a gradient- based solution method to solve the game by leveraging implicit differentiation, making it amenable to end-to-end pipelines.
Finally, we support the efficacy of the game and the solution method with extensive results.
RELATED WORK
The first related thread is machine unlearning, which focuses removing the influence of subset of data (referred to as the forget set) from a machine learning model. The unlearning approaches are divided into two classes. The first one is exact unlearning, which involves retraining the model on data excluding the forget set. The second one is approximate unlearning. The ideas behind approximate unlearning twofold. The first is track the influence of each training data the updates to model's 's weights, allowing for a rollback during unlearning (Bourtoule et al.) 2021 Graves et al. 20216 Chen et al.) 2022). The second is using loss function to capture the objectives of unlearning (e.g., removing the influence of the forget set while maintaining model utility) and modifying the
model weights to minimize the loss function (Guo et al.] 2020; Golatkar et al. 2020b Izzo al.)
2021b Warnecke et al. 2023 Chundawat et al. 2023 Jia et al. (023). The method proposed in this paper aligns with the second idea. Specifically. we design loss function that simulates an auditor who uses MIAs to evaluate the effectiveness of unlearning. By differentiating through the auditor's s optimization problem, we compute the gradients that reduce the auditor's utility, thus increasing the effectiveness of unlearning. Besides algorithmic developments, Jagielski et al. (2023) proposes measure to quantify the forgetting during training: [Thudi al. (2022b) takes formal analysis of the definition of approximation unlearning and propose methods to verify exact unlearning. Due to space
constraint, it is not feasible to provide a comprehensive review of all related studies. We refer the readers to the survey article by Nguyen et al. (2022) for more exhaustive discussion.
The second related line is membership inference attacks (MIA), Shokri et al. (2017 introduced MIAs, showing the privacy risks of machine learning models. Subsequently, different attack methods are proposed (Chen et al. 2021 Carlini et al.) 2022) Ye et al., 2022 Bertran et al. 2023). On the other hand, (Callin et al. (022) shows that existing criteria evaluate MIAS are limited in capturing real-world scenarios and propose more practical evaluation metrics. In addition, comprehensive evaluation frameworks and tools are developed (Murakonda & Shokri 2020; Song & Mittal| 2021). Finally, Nasr et al. (2018); Sharma et al. (2024) proposes a defense mechanism to counter MIAS from an adversarial perspective. Unlike these works, which relies on a purely NN-based minimax formulation limiting the integration of non-NN-based MIAs and existing unlearning methods, our Stackelberg game framework models the sequential nature of unlearning and auditing, enabling seamless incorporation of diverse, differentiable attack and defense strategies.
PRELIMINARIES
Machine Unlearning. Let D = (Z,, 3:) x; E X,,, EY} be a labeled dataset, where X' (resp. )) denote the feature (resp. label) space. The training, validation, and test sets are Dir. D Dval= and Dte. respectively. A machine learning (ML) algorithm is denoted by A. mapping from the joint space of features and labels x y to a hypothesis class. We refer to the model trained on the entire training set as the original model, i.e., 0,- = A(Drr).
2

---

Published as a conference paper at ICLR 2025
Let D 0,),),33 Drr represent forget set. The goal of machine unlearning is to remove the influence of Dj from the original model, resulting in unlearned model 0n (i.e., 0. ((O))) where U/ represents a machine unlearning algorithm. The unlearning algorithm may have access to other inputs (e.g., the validation set Duat) depending the problem settings. Let D, be the retain set, the subset of the training data excluding the forget set, i... D, Dr. \ Dy. The gold standard of machine unlearning O, A(D,), , a model trained on the retain set, excluding the influence of D We denote f, as the gold standard when comparing machine unlearning algorithms. Retraining is expensive, especially for deep neural networks. This motivates the development of efficient machine unlearning algorithms that satisfy the following conditions: 1) the influence of Dj is removed from the unlearned model, 2) the performance of the unlearned model is comparable to the performance of the original model, and 3) the computational costs (e.g.. running time) are lower compared to those incurred during retraining.
Membership Inference Attacks. A membership inference attack (MIA) aims to determine whether a data instance was used to train an ML model (Shokri et al. (017). An instance that was in the training set is called member, while one that was not in the training set is called a non-member Formally, given a target model 8, an attacker infers the membership of an instance (æ, y) based on the model's 's outputs (i.e., (o)r: and the label. The attacker does not have access to either the training data or the model parameters of the target model. Instead, he gathers proxy training and test sets and learns model 0 to mimic the behavior of the target model. Using the outputs of 0 on its own training and test data, the attacker acquires a labeled (member v.s. non-member) dataset, and then uses the labeled dataset to train a binary classifier for determining the membership of an instance.
We adapt the idea of MIA to determine whether the influence of the forget set still exists in an unlearned model 0.. Define an auditing set Dø.. = {(s), 1), (syr 0)} where s/ (resp. $
represents the outputs of the forget (resp. test) instances from the unlearned model, that is, sf So,, (2))) (resp- s So,, (x(+))
Here, the test instances serve as an empirical distribution for the unseen data. The outputs can be scalars, such as the ristance-wise cross-entropy losses. The outputs can also be the vectors of probabilities across the classes (Shokri et al.) 2017; Carlini et al.] (022). The labels "1" and "O" indicate members and non- members,,,, respectively. The MIA reduces to binary classification task on De aiming to differentiate the forget instances from the test ones based the outputs.
4 THE GAME MODEL
We model the machine unlearning problem as a Stackelberg game (SG) between an unlearner who deploys models as services, and an auditor who launches MIAs against the models. The key idea is to assess the effectiveness of an unlearning algorithm by measuring whether the auditor will succeed. In particular, the unlearning is considered effective when the auditor is unable to differentiate between the forget set from the test set based on their outputs from the unlearned model. The SG is played in sequential manner: the unlearner first deploys unlearned model, and then the auditor launches an MIA in response. Importantly, the advantage of first-mover endows the unlearner with the power to make a decision knowing that the auditor will play a best response (i.e., launching strong attack). We now formally define the models for both players.
4.1 THE AUDITOR'S S MODEL
We begin by defining the auditor's model. Suppose the unlearner has deployed an unlearned model 0,. Following standard setup (Shokri et al. 2017 Song & Mittal 2021). we assume that the auditor has black-box access to the model, allowing him to query the model, e.g., submitting data to the model and collecting the outputs. The auditor's goal is to determine whether the influence of the forget set still exists in the model based on the outputs. To achieve this, the auditor constructs an auditing set De., consisting of the model's 's outputs when passing the forget and test instances through the unlearning model 6, (see Section B for details about the auditing set). The auditor assesses the distinctiveness of the two sets with a binary classifier trained on the auditing set through cross validation.
3

---

Published as a conference paper at ICLR 2025
Let Uo be the auditor's utility function, quantifying the distinctiveness of the forget and test instances. Intuitively, a large Ug indicates that the outputs of the forget instances are highly differentiable from the outputs of the test instances, strong evidence that the influence of the forget set still exists in the unlearned model. We formulate the auditor's model as the following optimization problem
Ua(On; 0.) (Dg. ; where On E BB__
arg max M(D 0a)-
(I)
0g ER.
The auditing set Do. is divided into the training D and the validation Deef sets. The constraint encodes the process of learning binary classifier. The set Be, are the auditor's 's best-responses to the unlearner's decision 0,, that is, specific MIA that maximally differentiates the forget and test instances ( The function M is evaluation metric for the binary classifier dataset. The definition of M is flexible. One can use the accuracy to quantify the average performance of the classifier, where true positives are weighted equally with true negatives (Shokri al., 2017 Song & Mittal 202T). Alternatively, an average measure may not capture real privacy threats. Instead, ROC curve or true positive rates at specified false positive rates can be used for evaluation Carlini al. (2022).
The auditor's 's model exhibits a high degree of generality, unifying several advanced MIAs in the literature; this includes neural network- based attacks proposed by |Nasr et al.] (2018), quantile regression-basse attacks from Bertram et al. (2019), and prediction confidence-based attacks by [Song] Mittal (202T), etc. Under the formulation of Equation [ the mentioned attacks differ in 1) the hypothesis class H. of the binary classifier and 2) the objective function M. Notice the dependence of the auditor's 's best-response on O, (i.c., BB_,) arising from the unlearner's first-mover advantage. The unlearner utilizes this dependence to select unlearned model that limits the auditor's 's discriminative power, which we discuss next.
4.2 THE UNLEARNER'S MODEL
Next, we define the unlearner's model. Let C. represent the unlearner's cost function, which encompasses main objectives for unlearning. The first objective is maintain the utility of the model, ensuring that the unlearned model performs comparably (e.g.. in terms of predictive power) to the original model. To achieve this objective, we minimize loss function L(D,;; 9.)) computed on the retain set D,, following the principles of empirical risk minimization. All regularization terms are included in the loss function simplify notation. The second objective focuses on eliminating the influence of the forget set from the model being unlearned. We approach this objective adversarially by considering the auditor's 's utility M. In essence, smaller value of the auditor's utility indicates that the forget set is harder to be distinguished from the test set, providing strong evidence that the unlearning process effective.
Formally, the unleamer's 's optimization problem is to minimize the cost function below
Cu (Bu. 0z) = ((),: + a M( 0a).
(2)
The parameter E IR+ balances the loss L and the auditor's utility M. Depending on the specific setting, the cost function C, can be extended to incorporate additional objectives for unlearning. For instance, one specify that the unlearned model should perform poorly on the forget set (Graves) et al. 2021b); this can be achieved by minimizing an evaluation metric (e.g., likelihood) on the forget set. Also, several prrnmtti- techniques have been shown helpful for unlearning (Jia et al. 2023; ; one way to achieve this is by adding an ε regularization to the cost function.
4.3
THE STACKELBERG GAME
Now, with the unlearner and the auditor models in place, we formally define the Stackelberg Game (SG). The SG is to solve the following bivle optimization problem (Colson et al., 2007)
min 9. EH.
L(D,; (w) + M(Dral tu. 0a) s.t. 0a E Bo,
Auditor
For example, |Brtraa et al. (019) proposed quamtle-regression-bbeee MIA. In this case, the hest response the optimal model parameters for the regression.
4

---

Published as a conference paper at ICLR 2025
The objective function has two components: the first ensures generalization by minimizing loss the retain set, while the second quantifies privacy leakage by assessing the auditor's ' ability to differentiate between forget and test instances. A lower auditor utility indicates more effective unlearning, it reduces the distinguishability between the two sets. The hierarchical structure encodes the sequential order of the play, with the upper level corresponding to the unlearner's optimization problem and the lower level capturing the auditor's best-responses. During the unlearning, the unlearner needs to proactively consider the auditor's responses. This requires selecting an unlearning model where the influence of the forget set is erased, or from the auditor's perspective, the forget instances are indistinguishable from the test ones.
The key assumption of the SG is that if the forget set cannot be distinguished from the test set in terms of the effectiveness of an MIA-its influence is deemed eliminated from the unlearned model. We justify this assumption from three angles. First, one common way to measure forgetfulness is by assessing the accuracy of the unlearned model on the forget set (Graves al., 2021b, Chundawat ct al. 2023: Baumhauer et al.) 2022). This approach is grounded on the observation that machine learning models exhibit distinct performance between training data and unseen data. However, is important to note that accuracy the forget set does not necessarily correlate with forgetfulness, as there are inherently difficult (or easy) instances that result in low (or high) accuracy regardless of whether they were part of the training set (Carlini et al. 2022p. Secondly. MIAs have been used to study training data forgetting (Jagielski et al. 2023), demonstrating its utility in detecting residual traces of dataset. Finally, from an adversarial perspective. if a sophisticated attack like an MIA cannot differentiate the forget set from the test set, it is reasonable to expect that the influence of the forget set has been removed.
We solve the SG using gradient-based methods, allowing for easy integration into end-to-end training pipelines. Specifically, we use Implicit Function Theorem to differentiate through the auditor's optimization problem Equation ! obtaining the gradient of the auditor's s utility with respect to (w.r.) the unlearning model's ss weights, i.e., aM/00,, As a result, the SG becomes differentiable layer, compatible with the standard heward-backward computation. The solution methods will be detailed in the next section.
SG-UNLEARK, STACKELBERG GAME UNLEARN
In this section, we describe the algorithm for solving the SG. In general, it is NP-hard to find an optimal solution for the unlearner (Conitzer & Sandholm 2006). Instead, we focus on gradient-based algorithms find approximate solution, i.e., model parameter 0,, exhibiting good unlearning performance. The main technical challenge is computing the gradient of the auditor's utility w.r.t. the unlearning model's weights (i.e., 0M/ /O0)) which requires differentiation through the auditor's optimization problem. While the differentiation can be bypassed in some special cases, e.g., when the unlearner's hypothesis class is of linear regressions (Tong et al.) 22188), this is rarely applicable in the current setting given our primary focus on unlearning deep neural networks.
Our solution leverages both the Implicit Function Theorem (IFT) (Dontchev al. 2009) and tools from Differentiable Optimization (DO) to compute the gradients (Gould et al. 2016 Amos & Kolter 2017 Agrawal et al., (019b),, thereby rendering the SG differentiable layer seamlessly integrable into existing end-to-end pipelines.
We start by expanding the gradient of C₁2, W.r.t. On using the chain rule
ac OL(D,:0.) (M 0a) d0
añg ae
+ 09,
00,
08,
aD
The first term on the right-hand side can be easily computed using an automatic differentiation tool like PyTorch (Paszke al. (007). In essence, the computation involves passing D, through the unlearning model (i.e., B,,) in the forward pass, computing the loss L, and getting the gradients in the backward pass. The second term on the right is an expansion of êM /00,, using the chain rule; for clarity we omit the arguments of the functions. The gradient 0M/ 1000 is obtained by performing pass. Some evaluation metrics for binary classification, such as the 0-1 loss, AUC, recall, etc., are non-differentiable. Therefore, adhere to standard practices by employing a differenniable proxy for M, such as utilizing the logistic loss as a substitute for the 0-1 loss.
a standard
5

---

Published as a conference paper at ICLR 2025
Leveraging Implicit Function Theorem Computing the gradient a requires differentiation through the attacker's s optimization problem. The main challenge is the absence of an explicit function that maps D to 0a However, under certain regularity assumptions, one can derive an implicit mapping between D and 8, based the optimality conditions of the auditor's optimization problem (Gould et al., 2016). A concrete example is when the optimization problem convex, such learning support vector machind] In this case, the KKT conditions are necessary and sufficient conditions for the optimality, and it connects 0a with D through system of linear equations, i.e.,
J(D 0,
where f encapsulates the stationarity conditions, the primal and dual feasibility conditions, and the complementary slackness conditions (Boyd & Vandenberghe 2004). For illustration purposes, concrete example of the KKT conditions f for linear SVM is provided in Appendix A0 We apply IFT to the system of linear equations, resulting
a0,
Of ( ..)
Of(D 0a)
00,
aDg
For further insights into differentiating through an optimization problem using the implicit function theorem, recommend referring the lectures by Gould (2023).
Leveraging Differentiable Optimization In practice, we capitalize on tools from Differentiable Optimization (DO) to compute the gradients. Intuitively, implements IFT, as shown in Equation [6] for a given optimization problem. What we need to do is describing the auditor's optimization problem using specialized modeling language, c.g.. cvxpy (Diamond & Boyd (016)). We then use DO to transform this description into a differentiable layer. Subsequently, this differentiable layer is positioned atop the model undergoing unlearning. thereby establishing computational pathway from 0, to 0g. The pseudocode for this process is provided in Algorithm 1 This algorithm has a time complexity inherently describes an upper bound and n denotes the size of the attacker's optimization problem (i.... the number of variables and/or constraints). This cubic dependence stems from the matrix inversion in Equation When Equation Cartaaiioonaaaaaa admits specific structures, such when the auditor linear SVM, we leverage apth to exploit the linear structures of the KKT conditions and enhance efficiency. The corresponding results, denoted SG (Acc.), are presented in Table I with further discussion in Appendix A.II
consider DO software that
O(n²), where big-O notations
Algorithm 1 AGG-Unleann
1: Input: D,, Dy, Die and the original model 6,
2: Initialize: i 0, 0% = fo. scheduler 'y'
3: while < epoch do
4: Compute L(D,; 0%) on the retain set in forward pass
5: Update 8 + 0 n aL(D..8t,)
6:
Construct the auditing set Dgg from Dj and Dic
7:
Describe the auditor's optimization problem Equation I with cvxpy Convert the description to differentiable layer Audi torLayer Plug AuditorLayee into the computational graph
8:
9:
10:
Get the auditor's 's best response 0j AuditorLayer
Compute M Dual: 8i
11:
0M DDeg .0
Update 0++1 0-5 - nn²
805
13:
i i ++1
14:
end while
15:
Return: 0*,
This includes several state-of-the-art MIAs
(Bertran et al. 2023 Song Mittal 2021

---

Published as a conference paper at ICLR 2025
6 EXPERIMENTS
6.1 EXPERIMENT SETUP
We conduct experiments on both computer vision (CV) and natural language processing (NLP) datasets. For the CV tasks, we use the widely recognized image classification datasets CIFAR- 10, CIFAR-100, and SVHN (Krizhevsky et al. 2009 Netzer et al., 201T). The backhone model is ResNett-18 (He et al. leveraging the BERT Devlin 2018 model.
2016). For NLP tasks, we assess performance on the 20 Newsgroups dataset,
We explore two learning scenarios: random forgetting and class-wise forgeeiin In random forgetting. instances are sampled uniformly at random from all classes. In contrast, class-wise forgetting involves selecting all instances from a Specific class. For CIFAR-10 and CIFAR-100 the forget set consists of 10% of the entire training set, while the ratio reduced to 5% for SVHN. In all experiments, the attacker's 's optimization problem is formulated as binary classification task, where a linear support vector machine (SVM) is used to distinguish between forget and test instances.
The baseline methods we use for comparison with SG include Retrair Fine-Tume (FT) (Warnecke
al. 2021] Golatkar et al.) 2020a), Gradient Ascent (GA) (Graves et al. 2021a] Thudi et al. 2022a).
Influence Unlearning (IU) (Izzo et al. 202 la Koh & Liang 201 2017b), -s-srse (Jia et al. 2023)
Random Label (RL) (Haysse et al. |ll.a.2.0200 2020), Boundary Expansion (BE), Boundary Shrink (BS) (Chen
ct al. )023), SCRUB (Kurmanji et al. (202)) and DAU (Sharma et al. 2024). Further details on the bascline methods are provided in Section A.7 of the Appendix. For all methods, we use the SGD optimizer with weight decay of 5e-4 and momentum of 0.9. Other hyper-parametery
selected through the validation set. Specifically, we create a new auditing set. For each unlearning method, select the ryper-parameters that maximize the difference between the validation accuracy and the MIA accuracy on this new auditing set. This approach ensures that the model both generalizes well unseen data (high validation accuracy) and less vulnerable to the attacks (low MIA accuracy). The hy perrparraeeerss are listed in Table
9 in the Appendix. In addition, we propose an accelerated version of SG (Acc.) and the discussion is given in Appendix
6.2 EVALUATION METRICS
We evaluate SG and the baseline methods using metrics commonly adopted in prior studies (Bourtoule jet al., 2021 Jagielski et al., 2023 Dia et al.) 2023] Chundawat et al.) 2023). It is important to note that the test accuracy is evaluated on a subset of the test data that is separate from the one used for solving SG. Retain accuracy (Acc,) and test accuracy (Acote) are used to quantify model utility (Jial ct al. 2023). MIA accuracy, AUC and F1 score are the metrics to quantify the effectiveness of unlearning, all of which are estimated on the auditing set with 10-fold cross Carlini et al. (002)). An effective unlearning algorithm should result in MIA metrics that approach random guessing (0.5). Forget accuracy (Accy) and the absolute difference between the forget and test accuracy (lAccy- Acczel):
An effective unlearning algorithm should result in Accy being close to Accre- This indicates that the unlearned model no longer retains specific information about the forget data, as its performance on the forget set should be similar to its performance on unseen test data (Dre), reflecting the removal of the influence of Dj-
To gather additional statistical evidence regarding the effectiveness of unlearning, we collect the cross- entropy losses of the forget and test instances from the unlearned model into the empirical distributions C and L.e, respectivell. Next, we run a Kolmogoroy-Smirng statistics (KS Stat.) test to determine the distributions can be differentiated from each other. The KS statistic quantifies the differences between L, and Lte, where the p-value indicates whether the difference is significant (Massey Jr 1951). In addition to the KS statistics, we provide the Wasserstein distance (W. Dist.) between the empirical distributions of C and Lie. This complements the KS statistics and evaluates the unlearning performance in terms of the similarity between the losses.
Retraining the unlcarning model on Der Dy from scratch.
7

---

Published as a conference paper at ICLR 2025
6.3 RESULTS
The experimental results for random forgetting and class-wise forgetting are presented in Section 6.3.1 and 6.3.2 respectively. We consider retrain as the gold standard for evaluating unlearning algorithms: the closer to the metrics of retrain the more effective the algorithm. We highlight the closest metrics to retrain in bold.
6.3.1 RANDOM FORGETTING
We present the results for CIFAR-10, CIFAR- 100 and 20 NewsGroup in Table The results for SVHN datasets are provided in Appendix AA.3] SG achieves the best performance for most of the metrics, demonstrating its effectiveness in unlearning. Specifically, the KS statistic of SG is consistently lower than those of the other baselines, exhibiting an order of magnitude difference in the statisties for CIFAR-I0 compared to most baselines. Intuitively, ML models behave differently on training data compared to unseen data, and this difference is usually reflected in the corresponding losses (Carlimi et al. 2022). The small KS statistic of SG implies that the forget and test instances exhibit greater similarity in terms of the model's s behavior, although there is still a discernible difference between the losses. Another metric for measuring the similarity is the Wasserstein distance (W. Dist.). The baseline RL. achieves the lowest distance, although the difference with SG is not statistically significant. A visualization of the cross-entropy losses for the forget and test instances from one of the experiments is provided in Figure 4 The experiment results of TinyImageNet <Le & Yang 2013) and CelebA (Liu et al. 2018) are given in Table and [ We also consider the vision transformer model. The results of ViT model (Dosovitskiy 2020) CIFAR-10 are given in Table 5
Table 1: Experimental results (Meanga) on CIFAR-10 10, CIFAR-100 and 20 NewsGroup for random
forgetting. The highlighted metrics are the closest those of retraining, which are considered as the best performance compared with the other baselines. SG denotes the original method. SG (Acc.) uses hpth. SG + LiRA employs LiRA the attacker. SG (Acc.) RL sets L in Equation 4 as the objective of RL, with qpth as the attacker.
CIEAR.10
Actra
Jasy
Laaay Aesu
MIA
MIA
MIA
RTE (nin...
Merrann
0,909%
(19291 hocom
0.923%
0,0661
0.506%
0,50%4, 1,30
0,0255
0,0007
14.92
FT
0.9880
0.955 0.911
0,0737
0.5405
0.5057
0.45
GA
8,999 ownn
( 1999 eoo
0,0691
0.5504
0.361laxe7
0.6623
0,1400
0,27K2
0.18
IU
0.972h
0 0.922
0,0756
0.5.8
0.554% cess
0.10%0 166193
0.408
0.82
E--pame
0.9970 owen
B 1VV 099344...1
0.000
0.5501
0.5994.cem re47
D.68th..oo coss
0.1018...a
0 0,2244
0.96
RL
0.9983, oeen
0.9217
0.9810. ora.
0,0893
0.5217 ooer
0.3297 umss
0.5935
0,0996 m0s
0,1520
0.24
BE
2000
0.9304...oo
B 1999 pox0
0,0692
0.554 00
0.5639 cess
0.602% nesz
0.3412
0,2783 owce
0.27
BS
0 199 2oos
0,0683
0.558% 0072
0.577% ees7
0.0094y mos
0,1400
0 0.3072 oone
0.16
SCRUU
1.251...m
0 990% ex00
0.07100
0.535 653o
0.5699.007
0.10:%p
0.2483ymnm4 on14
1:30
DAE
0.9583 wes
0,009% wmm
0,931 ssus
0.0022
0.5184, enrs
0.5146,
0.6430, ress
0.0893, uwe7
0.1216 ownn
0.34
08900.. ao05
0,4111
0.5204 e554
0.5134
0.04
(Acc))
0.902
pon
0,5110,
0.5200..on
0.02
10887
SG (Aec3 RL
0,9968, wus
was
0.927
1.566,
ove
0.0231
0.520%
0.5230,
0.52%
0,095%,
6.10K2,
1.79
CIRAIC-100
Ace.
Aen.
Ancy
Anry - Ave.
MIA soc
MIA AUC
MIA
KS Sta
RTB onin. D
Recain
0,9996
0,0110
0.5184. cosr
0.5251a4003
0.5104
0,0200
0.0867uxxe
13.08
8,9996 swel
0.7158uae
2o06
0.2867
0.2833
0.977
0.66ll, esto
0.7601
0.7207
1,2219, 0
1,2583 once
0.20
0.39
0.997
19999 0.7026.0000
0,2933
0.7305
0,6950,
1.2612 0.4583
0.31
1999
0,2765
06738a ess1
0:399221es
0.3217am033
1. 1.155
0.86
Itl.
0.9065
1488
0.1818
0.580%
06152.1037
6.4000
0. 008067
0.7%
0.9995
07773umms
B a0e:
0:2823
0.765lures
6.7248,1045
1.2175 0.4940
0.23
BS
0.9995u
07160uams
0.2836
0.7651axes2
6.7259,.ess
1.2382 0,4963
0.39
SCRUB
0.709 09999)
(1999) heaau
0,2894
8.7015
0,7747,x093
0.7299
1.2290 0,4717amass
1.14
DAU
0 0800 1mm
0.135
0.5700
0.3816:
0.0006
0,1300
0.
13:35
SG
0.378
0 o.2
0
0.5520..eos
0.606l.x.0
0.098
1,5316
3,07
SG (Auu))
0 2.002 sud
0.1004
0.551 eaas
0.557 0.557
0.6192
0.1130
0.6176 owen
1.55
SG L.KA
00004
0.411 au
0.5983
0.6087
0.0274 uxea
1087 2w04
10.68
5O (Aoc) + RL.
6675% wed
1800% en
0.1416
5.5% eaer
0.000 mara
0.003% cens
0:3041 mnas
0.6191 ran
2.39
20 NewsGrrup
Ase,
Artta
Aesy
MIA soc.
MBA AUC
MIA
KS Stu.
W: Das
RTE (win..
Retrain
1.0000
085288
0.9224
0.0696
0.5355
0.5512
0.5531
0.1405
0.9925
40.8
FT
0.9999
08518
0.8055
0,8482
0.5672
0,6059
0.0230
0,2495
1.1894
20.2
GA
0.8481
00185
00500
0/0017
0.4995
0.4933
0.2704
00034
0.8800
1.0000
ILH575
0.1415
0.5676
0.6054
0.6614
27.9
0.9585
0.8295
0.1589
0.7123
0.7651
0.7148
0.5334
1,1402
21.2
1,0000
1.0000
1.0000
0,0000
0.0007
156
Another observation from the table is the inherent trade-off between model performance, measured by test accuracy, and the effectiivness of unlearning, measured by MIA accuracy. This trade-off has been documented in prior studies as a common challenge in unlearning tasks (Golatkar et al. 2020a: Bourtoule et al., 202T). Specifically, SG is more effective at unlearning the forget instances,
8

---

Published as a conference paper at ICLR 2025
as indicated by the highlighted MIA metries. However, this effectiveness comes cost to the test accuracy on CIFAR-10 and CIFAR- 100, a phenomenon observed in other unlearning techniques as
well (Jia al. 2023 Graves et al. 2021a). Despite this trade-off, the degradation in test accuracy
remains minimal.
6.3.2 CLASS-WISH FORGETTING
We use CIFAR-10 as the benchmark for class-wise forgetting, For results on other datasets and discussion, please refer to Appendix A.9
In random forgetting. instances are uniformly sampled across all classes, preserving the overall dataset distribution. In contrast, class-wise forgetting removes all instances from a specific class, resulting in significant distribution shift that makes forget data more detectable. The experimental results, presented in T highlight the metrics closest to retraining. However, all methods perform poorly on MIA-relat Arelted metrics, as the auditor can easily distinguish between forget and test instances due to the distinct distributional shifts caused by class-wise forgetting, Additionally, no single method consistently outperforms the others.
Table 2: Experimental results (Mean CIFAR-10 for class-wise forgetting. The highlighted metrics are the closest to those of retrain, which is considered as the best performance compared with the other baselines.
CIFAR-10
Als,
Autu
Auy
Jany Actral
MIA
MIA AUC
MIA
KS Stal.
W. Diet
RTE (nin..
Retnan
0,93333 ocea
0,00000
0.3335
0.9925
0.9980
0.993/
0.99020
9 9.5001
13:96
F7 GA
0,6043, .092
0.3183
0,5985 0.9915
0.7975 0.9915
019524 oen
1.16
0.84788..44
0.79422.e55
0.000
0.7935
0,9934.001
0.9095x.aw
0.993%.9915
(.920% 8557
15 29.2911,. 1o0e
0.53
IL
09339%
0.8644..4
0,0019
m9
0.8225
0.9972eaxe
0.9095
0.9972a.507
0.115 8::90
8.2574 6404
0.31
f fusporse
059972) cess
res
0.09
08311
0.9010
01908%%.oom som
0.901
0.920% 657s
2.588 mos
1.30
0.9996.co cens
0.9330,
a.000l..er
0.9999 som
0 e130
6.3989, arom
1.97
09710].012
0.5954)2c22
0.2477 ooa2
0.9964uuxe
0.9900
8.9964u,300
11.700ux 6our
0.32
09691a0e0:
0.2504 1.596A
0.9905
0.9985
0.9965
0.719 0572
5.0155
0.66
SCRUB
1,000% ros3
0,9209
0,000)..ce
10000ueee
10000uaee
1000
0.599% 0001
70, 70.9934;
3.47
0,0000 1,9056,
0.9056
0.9814
0.9902
aro
0.9696
5,2754, soe
0.84
6.3.3 THE EFFECT OF THE ATTACKER MODEL
Finally, we conduct a comparative study to understand the impact of adversarial modeling on the unlearning process, controlled by the parameter as defined in Equation [ We show the results for random forgetting and defer the results for class-wise forgetting to the appendix. In Figure [ we compare two cases where or is set to either 1 or O, denoted by SG-1 and SG-O respectively. The comparison is done across four metrics: 1) the test accuracy; 2) the MIA accuracy; 3) the defender's s utility, evaluated the test accuracy minus the MIA accuracy, which provides combined scalar value that measures both the performance of the unlearned model and the effectiveness of unlearning: 4) the Wasserstein distance between the empirical distributions of L and Lic. We show the averages over 10 experiments with different seeds, and 95% confidence intervals are displayed. The first observation is that the adversarial term (i.e., a M(Dgal 0.)) acts as a regularizer, improving the generalizability of the unlearned model. This observation supported by comparing the test accuracy of SG-1 and SG-O on CIFAR- 10 (top middle). Similar findings have been reported in Nasr et al. (2018). Another observation is that adversarial modeling limits the attacker's s ability differentiate between forget instances and test instances; this is demonstrated by the MIA accuracy on CIFAR- 100. The right-most column displays the Wasserstein distances between L and Lte. It is evident that the two losses are closer as result of adversarial modeling, especially for CIFAR-10 100 dataset. Additionally, the distances progressively decrease throughout the epochs, confirming the effectiveness of the gradient-based method.
In addition to the existence of attacker, we also investigate the strength of the attacker by changing o. We select the in large range of {0.05, 0.1, 0.25, 0.5, 1, 2). In Figure 2 in Appendix, we compare the performance regarding the test accuracy Accie The cross of the red dash line is the performance of the retrain model. We can find that SG is robust the attacker strength.
6.4 MIA SELECTION
To validate the generalization ability of SG, we select the MIA used in (Jia et al. and we evaluate SG according to (Jia al. 2023). The results given in Table show that SG is not
2023) as the attacker
9

---

Published as a conference paper at ICLR 2025
Deteende Unilet
Test Accurccy
MIA Accuneyy
Wassersiein Distance
S0-0
0.3
0.2
0.3
0.65
1.25
08.00
0.080
0.60
0.025
0.60
20 0.50
20
30
Epach
10
10
20
Figure 1: An ablation study to understand the impact of adversarial modeling on the process of unlearning; 1 and 0 corresponds to the cases with and without adversarial modeling, respectively. The results are the averages over 10 experiments with different seeds, and 95% confidence intervals are displayed. From the left the right: 1) the defender's utility, evaluated as the test accuracy Accie minus the MIA accuracy: 2) test accuracy; 3) MIA accuracy; 4) Wasserstein distance between the gross-entropy losses of the forget and test instances. Top row: CIFAR-10; Bottom row: CIFAR- 100. Epoch O: Original model.
sensitive to the MTA. Furthermore, we also consider the state-of-art MIA which is LiRA et al. 2022 as the attacker. The results are given in Table
(Carlini
Table 3: SG evaluated using the MIA attacker described in (Jia et al., 2023), comparing its performance against baseline models on the CIFAR-10 dataset under the random forgetting paradigm. For the metrics UA, MIA, RA, and TA, the value closest to the Retrain baseline is highlighted in bold.
CIFAR-10
UA (1 - Accy)
MIA
RA (Aoc,)
TA (Acr))
Avg. Gap (I)
RTE (min, })
Retrain
0.0807g.0047
0.1741
1.0000p.0001
0.91618.0024
24.66
FT
0.01 10p.0019
0.04066.com 8044
0.9983y.om0
0.9370g.om0
0.0555
1.58
GA
0.0056y.000
0.0119p.awx
0.9948 0002
0.9455
0.0680
0.31
IU
0.1751..0210
0.213%p.0170
0.8328p.0244
0.7813p.0265
0.1091
1.18
E-sparee
1.012
0.0433g.oors 00052
p.973
0.95499..01s
0.0661
1.82
RL
0. 0.0280p.00s7
0.1859p.nsss
0.9997a.0001
0.9408p.o012
0.0224
1,98
BE
0.0000,0000
0.00266,
1.0000,.0000
0.9535u.0015
0.0724
3.17
BS
0. 0.0048p.o0m 0007
0.01 0000
0.9947g.00m
0.9458p.oous
0.0684
1.41
SCRUB
0.0070n.005
0.0388p.0125
0.9959p.00s4
0.9422n.0028
0.0598
4.05
SG
0.0748p.00
0.1835
0.9990y.0131
0.9072p.018
0.0063
3.48
DISCUSSION
In this paper, we design an adversarial framework for addressing the problem of unlearning set data from machine learning model. Our approach focuses evaluating the effectiveness of unlearning from an adversarial perspective, leveraging membership inference attacks (MIAs) to detect any residual traces of the data within the model. The framework allows for proactive design of the unlearning algorithm, synthesizing two lines of have heretofore progressed in parallel. By using implicit differentiation techniques, we develop gradient- based algorithm for solving the game, making the framework easily integrable into existing end-to-end learning pipelines. We present empirical results support the efficacy of the framework and the algorithm. We believe our work can make a progress in trustworthy MI..
unlearning and MIAs-tha
10

---

Published as a conference paper at ICLR 2025
ETHIC STATEMENT
This work does not involve potential malicious or unintended uses, faimess considerations, privacy considerations, security considerations, crowd sourcing, research with human subjects.
REPRODUCIBILITY STATEMENT
We provide details reproduce our results in Appendix A.7 and A.8 We also provide pseudo-code in Algorithm and will release the code upon acceptance.
ACKNOWLEDGMENT
Z. Di and Y. Liu are partially supported by the NSF under grants IIS-2007951, IIS-2143895 and IIS-2040800. Y. Vorobeychik supported by the NSF through grants IIS-2214141 IIS- NSS-1905558, and as well as by the ARO under grant W911NF-25-10059 and the ONR under grant N00014-24-1-2633.
REFERENCES
A. Agrawal, B. Amos, Barratt, Boyd, Diamond, and Z. Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, 2019a.
Akshay Agrawal, Brandon Amos, Shane T. Barratt, Stephen P. Boyd, Steven Diamond, and J. Zico Kolter. Differentiable convex optimization layers. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), pp. 9558-9570, 2019b.
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization layer in neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 136-145. PMLR, 2017.
Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer. Machine unlearning: Linear filtration for logit-based classifiers. Machine Learning, 111(9):2033-322 2022.
Theo Bertram, Elie Bursztein, Stephanie Caro, Hubert Chao, Rutledge Chin Feman, Peter Fleischer, Albin Gustafsson, Jess Hemerly, Chris Hibbert, Luca Invernizzi, et al. Five years the right to be forgotten. In Proceedings of the ACM Conference Computer and Communications Security (CCS), pp. 959-972 2019.
Martin Bertran, Shuai Tang, Michael Kearns, Jamie Morgenstern, Aaron Roth, and Zhiwei Steven Wu. Scalable membership inference attacks via quantile regression. CoRR, abs/2307.00 0.3694 2023. URL https:://oi..o////////// 48550//00..555///////../ 00.......3..... 03694
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette- CChoo, Hengrui Jia, Adelin Travers. Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. Proceedings of IEEE Symposium Security and Privacy (S&P), pp. 141-159. IEEE, 2021.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In Proceedings of the 39th IEEE Symposium on Security and Privacy (S&P), pp. 1897-1914. IEEE, 2022.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Yongdae Kim, Jong Kim, Giovanni Vigna, and Flaine Shi (eds.), Proceedings of the ACM Conference on Computer and Communications (CCS), pp. 896-911. ACM, 2021.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang Graph unlearning. Proceedings of the ACM Conference on Computer and Communications Security (CCS), pp. 499-513, 2022.
11

---

Published as a conference paper at ICLR 2025
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7766-7775, 2023.
Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Proceedings of Thirty-Seventh Conference on Artificial Intelligence (AAAI), pp. 7210-7217, 2023.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153:235-256, 2007.
Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy to commit to. In Joan Feigenbaum, John C.-L. Chuang, and David M. Pennock (eds.), Proceedings 7th ACM Conference on Electronic Commerce (EC), pp. $2-90. ACM, 2006.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Steven Diamond and Stephen Boyd. Cvxpy: A python-embedded modeling language for convex optimization. The Journal of Machine Learning Research, 17(1):2909-299 -913,, 2016.
Asen L Dontchev, R Tyreell Rockafellar, and R Tyrrell Rockafellar. Implicit functions and solution mappings: A view from variational analysis, volume 616. Springer, 2009.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929) 2020.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net:
Selective forgetting in deep networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9301-9309, 2020b.
Stephen Gould.
Lecture notes on differentiable optimisation
deep learning.
CoRR, 2023.
URL https:// /ttps///uerrs.ee cecs anu. edu. wu/-s/go.dd/ppppzs/
pdf
Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application bi-level optimization. CoRR, abs/ bobs/1607 1677 .0447,, 2016.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 11516-11524, 2021a.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh, Amnesiac machine learning. In Proceedings of the 33rd Conference Artificial Intelligence (AAAI), pp. 11516-11524, AAAI Press, 2021b.
Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pp. 3832-3842. PMLR, 2020.
Tomohiro Hayase, Suguru Yasutomi, and Takashi Katoh. Selective forgetting of deep networks at finer level than samples. arXiv preprint arXiv:2012.11845 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2016-2016. PMLR, 2021a.
12

---

Published as a conference paper at ICLR 2025
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics (AISTATS), volume 130, pp. 2008-2016, 2021b.
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito. Katherine Lee, Nicholas Carlini, Eric Wallace. Shuang Song, Abhradeep Guha Thakurta, Nicolas Papemot, and Chiyuan Zhang. Measuring forgetting of memorized training examples. In Proceedings the I 11th International Conference on Learning Representations (ICLR), 2023.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsification simplify machine unlearning. CoRR, abs/2304.04934 2023. wtt.l:ttt////tt wttp://i..rr.1............
arXiv 2304, 04934
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pp. 1885-1894, 2017a.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1894. PMLR, 2017b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou, Towards unbounded machine unlearning. Advances Neural Information Processing Systems, 36, 2024.
Yann and Xuan Yang. Tiny imagenet visual recognition challenge. 231N, 7(7):3, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
Frank J Massey Jr. The kolmogoroy-smirrov test for goodness of fit. Journal of the American statistical Association, 46(253):68-78 1951.
Sasi Kumar Murakonda and Reza Shokri. MI privacy meter: Aiding regulatory compliance by quantifying the privacy risks machine learning. CoRR, 0es/20077099333 2020.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In Proceedings of the ACM Conference on Computer and Communications Security (CCS), pp. 634-646, 2018
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. CoRR, 0is/2209.02299, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Rohan Sharma, Shijie Zhou, Kaiyi Ji, and Changyou Chen. Discriminative adversarial unlearning. arXiv preprint arXiv:2402.06864 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In Proceedings of the IEEE Symposium on Security and Privacy (S&P), pp. 3-18. IEEE, 2017.
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks machine learning models. In Proceedings of the 30th USENIX Security Symposium (USENIX), pp. 2615-2632, 2021.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (FuroS&P), pp. 303-319. IEFF, 2022a.
13

---

Published as a conference paper at ICLR 2025
Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In Kevin R. B. Butler and Kurt Thomas (eds.), Proceedings of the Thirty-first USENIX Security Symposium (USENIX), pp. 4007-4022, 2022b.
Liang Tong, Sixie Yu, Scott Alfeld, and Yevgeniy Vorobeychik. Adversarial regression with multiple learners. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pp. 4953 2018.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprini arXiv:2108.11577 2021.
Alexander Warnece, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. In Proceedings of the 30th Annual Network and Distributed System Security Symposium (NDSS)) 2023.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the Conference on Computer and Communications Security (CCS), pp. 1093-3106. ACM, 2022.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
The Appendix is organized as follows:
Section
the description of all the notation we use in this paper.
Section
A.2
the experiment results of ViT model and CIFAR-10 and CIFAR-100.
Section
A.3 3
the experiment results for random forgetting SVHN dataset.
Section
A.4
the experiment results for random forgetting on TinylmageNet dataset.
Section
A.5
the experiment results of the practical scenario on CelebA dataset.
Section
A.6
the cross-entropy loss distribution the testing and forgetting instances.
Section
the baseline method we use in this paper and their settings.
Section
A.8
the experiment details and hyper-ppanee syper-parameters.
Section
A.9
the experiment results of class-wiss forgetting
Section
A. 10
the detailed explanation of our problem formulation and how we solve the problem.
the implementation details about the accelerated version of the proposed method and the performance comparison.
Section 11
Section
A.12
the experiment results of sequence unlearning
A.1
NOTATION TABLE
Table 4: A summary of the notations used in the paper
Notation
Mcaning
D ((F Hi) (Ii, vi) N.Y
A dataset
One data point where T; is the feature while ys is the label
The feature space and the label space
Dj, Dres Doats Der, Dr
The forgetting, testing, validation, training, and retain
(x v
One data point from the forget set D,
A machine learning algorithm
u
A machine unlearning algorithm
0.
The original model, i.e., A(D,,)
On
The unlearned model, i.e., (1.)))
Or
The retrained model, i.e., A(D,)
De
The auditing dataset for membership inference attack The output of a forget instance in the auditing dataset Dp from On. The output of testing instance in the auditing dataset De., from O..
sfe
Der Deal
The training and validation split of De..
C
The unlearner's cost function
Ba..
The auditor's ' best response given an unlcarning model 0,,
Un
The utility function of the auditor
Ha
The hypothesis class of the auditor The hypothesis class of the unlearnr
Ha
The trade-off factor as defined in the unlearner's cost function Equation
A.2 CIFAR-10 AND CIFAR- 100 DATASET
A.2.1 VIT RESULTS
In addition to the models based on convolution layer, we also test our method on the transformer-bar based models. The model we use is ViT and the results are given in Table
15

---

Published as a conference paper at ICLR 2025
Table 5: Experimental results (Mean on CIFAR-10 for random forgetting using VIT. The high- lighted metrics are the closest to those of retraining, which is considered as the best performance compared with the other baselines.
CIFAR-10 10
Ace,
Jen,
Jcsy
Veeyy Aoc..
MBA acc.
MIA AUC
MIN PI
KS Stat.
W. Dist
RTB (mia.
Retmon
08886 som
0.24277
0.7483a
0.000
0.5000s over
0,5000 rxes
0.59Me
0.19%
0.0.004 sars
200. 90s
18863
0.688
0.8675
Bussez
08673 086774.9915
0.0988am4
0.508
00000
0.30
0.5157
Neavos
0.8322
0.5000
0.104
A.2.2 THE EFFECT OF ATTACK STRENGTH
MIA Accuracy
Forget Accuracy
KS Statistic
0.05
0.50
0.075
0.1
0.920
0.25
0.48
0.050
0.915
0.46
0.025
0.90
0.91
0.92
0.93
0.90
0.91
0.92
0.93
0.90
0.91
0.92
0.93
0.750
0.10
0.54
0.725
0.05
0.52
0.700
0.64
0.66
0.68
0.70
0.64
0.66
0.68
0.70
0.64
0.66
0.68
0.70
Test Accuracy
Figure 2: Experiments with different values of the trade-off parameter a. We consider 7 values {0.05, 0.1, 0.25, 0.5, 1, 2, 5). Each dot represents batch of random experiments with the same a. The coordinates of a dot are the corresponding metrics averaged over the 5 runs. Top row: CIFAR-10; Bottom row: CIFAR-100.
A.3 SVHN DATASET
The results of SVHN dataset on random forgetting are given in Table The comparison of SG on SVHN for random forgetting with and without attacker is illustrated in Figure 3
Table 6: Experimental results (Mean SVHN for random forgetting. The highlighted metrics the closest to those of retraining, which is considered as the best performance compared with the other baselines.
SVHN
Ans
Artu
Awy
Auyy Aetsv
MIA
MIA AUC
MIA
KS Stal.
W. Dist.
RTE (min.
Retran
0,9959,
0,9600s
0.9534p
BU0066
0,5298
0.5422aaam
0.630% 0.514%
0.06866
20.46
FT 0.9991.cem
0,9876 0.71
0.2567
0.5372
0.5t92.umm
0.5523
1.0613, eser
0.1745
1.55
GA
099544 cont
0.641a 2002
0.9918
00000
0.5191
0.541laaxc
0.5500%.117
IL0867a eass
0.1475
0.97
IU
099776.1097
0:8817a
0,9050 mma3
00235
0.5323gamm
0.55%0ua07
0.546%.555
IU047 8217
0.1407axes0
6.41
4y 4ysppane
0.9578 ress
09199a cess
0,9290 weds
00107
0.5457
015005uuze
0.53477.554
0..96...11
0.115 eese
reaz
ress
oens
0.5211; unce
0.5411mmar
0.5144,175
11107%
0,864,, ross
19995
0.963,2002
0.9955
0.520%
0.5441uaa
0.5555uar
1.101 6100
0.1528
reaz
0.9441..00
0,9952
0.5323
0.550%
0.5094.1176
1109944
0.140% coss
neca
6.227
0.5431
0.950)
0.0855usen
A.4 TINYIMAGENET DATASET
The results of Imayema dataset random forgetting are given in Table
16

---

Published as a conference paper at ICLR 2025
Table 7: Experimental results (Mean on TinylmageNet for random forgetting. The highlighted metrics are the closest to those of retraining, which is considered as the best performance compared with the other baselines.
TimylmageNet
Ace-
Aocie
Aoe
|Abey Aec.
MIA
MIA AUC
MIA
Stat
KIE
Retran
0.877
0.5907
0.5057uams
0,0910
0.547
0.5077a
0,4600
237.12
FT
0 a.00
0.0338
0.540%
0.5836
0.0077.xes
0.9017ammms
65.07
GA
8.5966ummi
a18066.ave
0.2000
0.59%m.co
0.0022
0.6619.1ea
0. 19688
0.9195
12:40
IU
0.350
0:066uumu
01850) aem
0,2260
0.6500 eaas
0.670%.
0.21 R0use:
0,9684
6.73
4y-paase
0b15fuans
0003
0,800% eeas
0.5678
00000
0.223
101
RL.
(1995 sen
00000
85280 cmso
0.8702. enis
0.475% rear
1661 cms
0,3304 oea
0891%, nur
0:3477
8621 formo
0.67%% cess
11.3
0.5775
0.8222aam ara
0.2457
0.6333 exas
0.5818
0.2415
1.0023
8.67
0.900
vos
0,5822 0014
0,080)
0.446 ores
Figure 3: An ablation study to understand the impact of adversarial modeling on the process of unlearning; and = O corresponds to the cases with and without adversarial modeling, respectively. The results are the averages over 10 experiments with different seeds, and 95% confidence intervals are displayed. From the left to the right: 1) the defender's 's utility, evaluated as the test accuracy Accie minus the MIA accuracy; 2) test accuracy; 3) MIA accuracy; 4) Wasserstein distance between the cross-entropy losses of the forget and test instances.
Defender Ualley
Toe
MIA
Wasamdzin Dieanco
0.55
504
0.4
60.
0.3
3.99
0.30
0.50
0.30
3.540
0.94
0.0
30
Epech
10
A.5 CELEBA DATASET
We select CelebA dataset as the practical scenario. The model is to classify whether the human is smiling or not while the unlearning goal is to forget the identities. The results of CelebA dataset are given Table 8
Table Experimental results (Meanad) on CelebA. The highlighted metries are the closest those of retraining, which is considered as the best performance compared with the other baselines.
CeleA
Alec
Ansy
Vaasy
MIA
MIA
Siat.
Dist.
STE unin.
Retmin
0,5087a 199
0.0026
0.5123
05100
0.0885
0.028%
0.0866
$3.70
F7
0:9361aseus
DJ0063
0.5058
0.5070uaen
B6180usar
1.013..0mm
0.015kues22
3,43
GA
0964622041
019216y.004
0.9480
0.0204
0.52440as
0.22%
2,43
07710
015040%
00125
0.503
0.9074
0.28 05556h
n.28
00277a
5.12
R1.
0.977%aees
00122
0.9056
n0188 06230arr
00345..om
2.65
BE
0.9400 ..2
00096
0.908)
0.177
1.929%
0.8546...
3.50
BS
0.9444....
0.929
0,9466 awas
000955
0,5045
0,513%uees
0.6287uamx
(1027%) mam
0.0653
1,07
0,0222, 1.94
cear
0.9288 ce1s
0.0866
0.5189 ueco
0.5003....
0.010R.com
9.74
A.6 LOSS DISTRIBUTIONS
visualization of the cross-entropy losses of the forget and test instances is shown in Figure
A.7 BASELINE METHODS
The first baseline is retraining, where the unlearned model is obtained by training on the retain set from scratch. We aim to develop unlearning algorithms that the metrics they produce are as closely aligned with those of the retraining as possible.
Retrain:
Fine- Tine-Tuning (FT): As the second baseline, FT continues to train the original model on the retain set for a few epochs. This a standard baseline used in various prior research (Graves et al. 20216 Warnecke al. 2023
17

---

Published as a conference paper at ICLR 2025
Figure 4: The distributions of the cross-entren ents-ey losses for the forget and test instances from the unleamed models. The y-axis in log scale for better visualization. From the first to the last figure, they are random forgetting on CIFAR-10 CIFAR- 100, SVHN and class-wise forgetting on CIFAR- 10, CIFAR- 100, SVHN.
18

---

Published as a conference paper at ICLR 2025
Table 9: The hyper-parameter for the baseline method and SG used in this paper.
Parameters
Retrain
FT
GA
f
fp-spase
RI.
BS
SCRUB
DAU
Leaming rate Num. of epuch
lc-2
5c-2
le-3
le-2
le-2
lc-5
le-5
5e-4
[le-2, 2c-2]
lc-2
160
30
10
[]
7
Se-4
X
X
X
10
X
T
X
X
X
X
Decay epochs
X
X
X
X
X
X
X
K
[3.5.
91
X
X
X
X
X
X
Attacker
X
Gradient Ascent (GA): This baseline takes the original model as the starting point and runs a few epochs of gradient ascent on the forget set Dy. The intuition is to disrupt the model's generalizability on Dj Graves et al., 2021b). Another name of GA is NegGrad (Kurmanji et al. (024)..
Influence Unlearning (IU): This baseline uses Influence Function to estimate the updates required for model's weights as a result of removing the forget set from the training data (Izzo et al. 2021b
Koh Liang 2017a).
spspase:: This baseline integrates & norm-based sparse penalty into machine unlearning loss
Dia et al. (203).
Random Label (RL):
This baseline trains the original model on the retain and the forgetting set D, whose labels are random to make the model unlearn Dj while keep the model capability as much as possible.
Boundary Expansion:
This baseline proposes neighbor searching method to identify the nearest but incorrect class labels to guide the way of boundary shifting.
Boundary Shrink: This baseline artificially assigns forgetting samples an extra shadow class of the original model Chen et al. (2023).
SCRUB: This baseline achieve MU by using a teacher model student model Kurmanji et al. (202).
DAU: We follow the official implementation of (Sharma et al. 2024) on the CIFAR-10 and
CIFAR-100 dataset for random forgetting.
A.8 EXPERIMENT DETAILS
The hyperparameters used for SG and the basclines are in Table y The losses for the retraining baseline across the epochs are displayed in Figure We run all the experiments using Py PyTrcch 1.12 on NVIDIA A5000 GPUs and AMD EPYC 7513 32-Core Processor.
A.9 CLASS-WISE FORGETTING
The results of SVHN dataset on classwise forgetting are given in Table 13 Regarding non-iid samples, classwise forgetting represents extreme scenario in which the model is tasked with forgetting entire class from the dataset. The results for classwise forgetting are reported in Tables and [3] As noted in prior unlearning literature (Chen et al. 2023 Jia et al. 2023). classwise forgetting exhibits consistent trends, where the forgetting accuracy and MIA metrics reach This is because forgetting entire class from the training set is straightforward and results in clear distinction from the (SOTA) method that dominates across
remaining data. Importantly, there is no clear state-of-the all metrics for classwise forgetting, and baseline methods generally perform similarly. Furthermore. the performance gap between SG and the best metrics is minimal. For instance, the differences in MIA metrics between SG and the gest-performing approach are less than 1% in both Tables 2 and
A. AN EXAMPLE OF THE CONDITION IN EQUATION 5
In this section, we provide a concrete example of the KKT conditions for linear support vector machines (SVM). As described in Section I the KKT conditions are key to relating the attacker's model parameters, denoted 0a, with the auditing set De.. which allows to derive the gradient D0./ODo.. The conditions be similarly derived for any model where the learning problem is convex. To simplify the notations, use { 9: 1
represent Do A standard formulation of
19

---

Published as a conference paper at ICLR 2025
1.5
Loss
1.0
0
20
40
60
80
100
Epoch
1.0
Loss
0.0
50
100 Epoch
150
200
1.5
Loss
1.0
0.5
0.0
20
100
Epoch
Figure 5: The training loss for the retrain baseline. For CIFAR10 and CIFAR100, the learning rate is multiplied by 0.1 when epoch is at 60, 120, 160; for SVHN, the same multiplication is done at epoch 60, 120. Top to bottom: CIFARIO, CIFARI00, SVHN.
Table 10: Experimental results (Mean on SVHN for classwise forgetting. The highlighted metrics are the closest those retraining, which is considered as the best performance compared with the other baselines.
SVHN
Ane.
Arti
Anty
Javy Ast.,
MIA ave.
MEA AUC
MIA
KS Stal.
W. Dist.
RTE (mti., 13
Retran
0,99933
0.9039
0,00000uace
0.9639
(19950k
01998 2001
0.595 la coos
0,590% e033
8.692in
20.46
0.9978
1.0622..co
cas
0.9946,
2,9220
0,9414..eo>
0.9164
0.9998....
corz
16.4834,
0,8014
nore
0.99%...0
10000. 2900
19921.am
e012
2012
$.5978,
0.9880;
0.9566,
0.634
0.9976,
eero
4.381
00986422003
0.9557
0.310%
0.6428
0.9995.2202
0.9976.com
0.9072
4.42500.330
SCRUB
0.9010
1000..00
0.9616
(1999%
1.0000
0.999%
0.998%
24.4590,
50
0.97
O 9600
0.9601
(199%
99944,
0.99%%
0.9907,
5.0188
5.90
the linear SVM is as follows
min R
|10al" 12
2
s.t. y: (0 x. +
b) 2 1,
where b is the bias term. The standard form is typically formulated as minimization problem, the attacker is to maximize V - 110, 2 Equation is a convex program, and the optimal solution
(i.e., 0% and b²) is characterized by the KKT conditions. The Lagrangian of the above is follows
20

---

Published as a conference paper at ICLR 2025
Table 11: The attack time for each epoch between evxpylayers and gpth on different datasets.
CIFAR-10
CIFAR-100
TinylmageNet
CelebA
evxplayers apth
31.92s
70.13s
24.39s
173.04s
7.44s
7.97s
4.68s
15.09s
where 0ş 2 0 are the Lagrantian multipliers:
T(0 b, 0;) = =2 2110a
0i
(yi (0 24 1).
(8)
Following sandard procedures (Boyd & Vandenberghe 2004), the KKT conditions are as folllows
0a
a,yi =
f( f(Do 0a)
Onty
V1 (0 x; + b) 2 1
a 2 0, Vi
(0 x; b) - 1) = 0, Vi
which implicitly define function between 0g and the data Do (zi, y4 ) In practice describe the optimization problem Equation Jusing cvxpy (Diamond & Boyd 2016). Then, we employ an off-the-shelf package called evxpylayers (Agrawal et al. 2019b automatically derive the KKT conditions and compute the gradient 00./DDg..
A.11 THE ACCELERATED SG
To accelerate the SG performance, particular setting discussed in our work involves the MIA auditor using a linear SVM as the classifier, as detailed in Appendix [A.10] This specific problem be formulated Due to the linear KKT condition the decision variables, we leverage a specialized solver called apth (Amos & Kolter 2017) instead of using more generic convex programming solver like cvxpylayer (Agrawal et al. 2019a). The qpth solver takes advantage of the special structure of QPs, specifically that the KKT conditions can be expressed linear system equations, allowing for more efficient computation. This tailored approach significantly reduces the computational burden compared solving general convex programming problem shown in Table !
convex programming problem known Quadratic Programming (QP).
In addition, we compare the attack time and the accelerated time relative to retraining. The results are given Tables and 12
Table 12: The speed comparison against retraining across various datasets between evxpylayers and qpth. The percentage in parentheses represents the ratio relative to Retrain.
CIFAR-10
CIFAR-100
Tiny
CelebA
Retrain
14.92s
13.08s
33.70s
235.68s
cvxplayers
1.47s (9.85%)
3.07s (23.47%)
15,20s (45.10%)
127.92s (54.04%)
qpth
0.88 (5.90%)
1.55 (11.85%)
9.74 (28.90%)
13.36 (5.65%)
A.12 SEQUENCE UNLEARNING
In addition to one-time unlearning, we also explore sequential unlearning to enhance the ro- bustness of SG. The unlearning scenario adopt is random forgetting. Specifically, select
21

---

Published as a conference paper at ICLR 2025
10% of the training data, Dor, as the forgetting set D, and divide it into five disjoint subsets: {DJ)) D D D
D Each subset is used sequentially for unlearning. For instance, the first model, 0 is unlearned from the original model, 0o, using D Subsequently, the second model, 8 is unlearned from of using ) and so on. We report our experimental results on CIFAR-10. 10.
Table 13: Experimental results (Meanga) on CIFAR-10 for sequence forgetting. The highlighted metrics are the closest to those of retraining, which is considered as the best performance compared with the other baselines.
Part
Are,
Aces
Arny
Mony Aocwl
MIA
MIA AUC
MIA
KS Stat.
Dist.
Retrain
0.9999s.0000
0.9167
0.50
0.0127
0.1895g.0010
0,5045
0. 0,5066
0.0543
FT
0.9984,
0.92876.0019
0.99236.0017
0.0636
0,5497 cos9
a.5681
0.6824p.004
0.1 0.1180p.000
0.2801
GA
0,9997c.0000
0.99276.0020
1.0000
0.0673
0.5677%..m12
0.59466u.00
0.6064102044
0.1703u
0.307 40.002
IU
0.92236.0021 0090
000000
0.0677
0.56576.002
0.5971a
0.6053000111
0.1650uee00
0.3092a.g0ss
fy-spanee
0,8893..013
0.8723p..0tt
0.0116
0.49822..0s 0004
0.4822au117
0.4873.1117
0.0472...au 0.0820u.0oma
RL
0.9986 roo1
0.9257
0.0044
0,4790 0094
0.3016
0.61
161170
0 0.3600
0.1287
BS
0.9997,.0000
0.9322%..00 0015
100000..0000
0.0677
0.5675
0.6024000100
0.6960l..002
0.1703a
0.9076u.000
SALUN
0. 0.90
0.9274g.0024
0.9253
0.0041
0.479%..0127
0. 0.2666611014
0.61 1617990945
0 36440000122
0.1 0.1378u.0148
SG
0.988 0084
1.8851
0.9160..os 0154
0,0309
0.5027 0018
0.5115
0.0533 0.65
0. 0.1078 osss
Part 2
Ace,
Actie
Acyy
Aaay Accue
MIA
MIA AUC
MIA Fl
KS Stat.
W. Dist.
Retrain
00900
0.9143
0.9070g.0010
0.0073
0.50077..009
0.4
0.3844
0.0650
0,07624.0416
FT
0.9084c.000t
0.93 1g.0000
0.9930
0.0616
0.5483
0.5630mmm57
0.1080, 0.6806
0.256%
GA
0.9984, 10os
0.9299%.o0 001s
0.9987
0.0688
0.5667 0.058
0.5856u.0044
0.6932
0.1487a0m
0.2955..00
IU
0.9984c.com ross
0.92877. 0010
0000
0.0696
0.5675 0016
0.5917amm
0.151 0001
0.299% as0
1i-spame
0.81845.0011
0.8173p.0041
0.
0.7957
0.0216
0.49992.0155
0.505400155
0.4325
0.06077..007
0.0797o..0ss
RL
0.92788..007 0.9984c.0003
0.9343
0.0065
0,4735a.001
0.3138uunzo
0 0.3457u0000
0.1231
BS
0.9995c.com
0.9328c.omnz 00112
..000....0
0.0672
0.5667c.mm
0.5995p.mts
0.6950..006
0.1627m.000
0.3056
SALUN
0.9983, (000
0.9172
0.9290g.0022
0.0018
0,4867
0. 0.3077a00s
0.62360000
0 0.3473
0.13720.0104
SG
0. 0.9954c..mn
0. 0.8850
0.0315
0,5042 0016
0.6590
0,0790
0.103 0.103In.arrs
Part 3
Act.
Acts.
Acey
Aoey
Ancuel
MIA acc.
MIA AUC
MIA
KS Stat.
W. Dist.
Retrain
1.0000; roco
0.9154g.oom 0016
0.9103
0.0051
0.4952 0oss
0.4178 0.4937
0.0680,
0.0532,
PT
0.998422.001
0.9308p.0022
0.9000
0.05V7
0.5510p..0zz
0.5648u0117
17681700
D. 10270001
0.27276.0154
GA
0.9984c.com
0.9208s.001
0.9970..mm
0.0662
0.569%
0.5933uut
0.695%
0.1647aam
0.2802n.com
0.9985c.0003
0.9970...0m
0.0656
0.5700s..ss
0.5973mmmta
1.696l
0.1630ussst
0.2794n.2214
Syspuaee
0.823 roso
0.8193s.0015
0.7970. 00e5
0.0223
0.5065 0053
0.521
0.3560%
ooea
0.0653
0.0711 0991
RL.
0.9985,
0.9261
0.92800.0004
0.0019
0.4875
0.3752a.03
0.5544
0. 0.33830.000
0.1 1..3999.ooo
SALUN
0.9085c.0002
0.92633.0012
0.9353
0,0070
0,4900ls.0m7
0,309
6266
0.3317ua
1.12866.112
BS
0.9984s,0022
0.92870.0019
0,9923
0.0636
0.54976.0052
0.568lauz2e
0.682
0.1180u
0,280 2..00.
SG
0.9814
0. 6.889%..04
0 0.
0.0255
0.5013 0.5001
0.621900001
0.0857
O. 0.10190.044
Part 4
Act,
Aces.
Arty
Many
Arcee
MIA acc.
MIA AUC
MIA
KS Stat.
W. Dist.
Retrain
0.9999
0.9140k..002
0.9133
0.0007
0.5102g.
0.4959 0.5137
0.0747uuuu
0.0936
FT
0.9984;.000 0001
0,9286, 0020
0.9910,..0s
0.0624
0.5525 0058
0.5758
0.1147 0.6835
0,2896,,007
GA
0.99845.0000
0.929%2.0011
0.W077000000
0.0679
0.5732a.00
0.5915m0072
0.68770uu200
0. 14143uua
0,2902a.a1se
IU
0.9981c.0001
0.9303,.0022
0022
0.9053,0000
0.0680
0.5717c.0056
0.59660.000
0.6971n004s
0.1450muams
0.2901oaa10
fy-spure
0.877c.00000
0.8339..0055
0.8177
0.0032
0,49661s 00771
0.4851 amns
0.5910hu0m2
0.0577ua133
0.0606u.2077
RL
0. 0.9984c.c002
0.9268s
0.9357
0.0089
0.4925 0013
0.3360m.m1
0.629
0. 0.2990,
0,131 n00s
BS
0.9995,.0001
0.9309)
0011
0.9987
0.0678
0.5698, 004s
0.6330poous7
0.6970
0.1793
0.311%
SALUN
0.9982, roos
0.9242 0937
0.93236 01
0.0081
0.48676.000 0056
0.3332o0130
0.6114ooxu
0 0.3200..cmm
0.1263oama
SG
0.9845,
0,8836
0. 0.9153
0.0317
0,5118
0.513%
0.6266,
0.0807
0.1018,
Part 5
Acc.
Acti
Acey
Aacy-
Ancue
MIA acc.
MIA AUC
MIA
KS Stat.
W. Dist.
Retrain
0.999 roon
0.9123
0.90477
0.0076
0.5035 mo9e
0,5043u
0.0522m
0.0577a.m9
FT
0,9986 roon
0.9301a.001 0012
0,9883
0.0582
0.5540c 0017
0.1133 0.6852 0.564Huums
0.280%,.cau
GA
0. 0.998Sc.0002
0.9298p.ux12
0.99832.0112
0.0685
0.5695
0.5809
0.694880000
0. 0.1467ua077
0.3055u.a37
IU fy-spanse
0.9986k
0.92866. 0036
0.9987a.000
0.0701
0.57177
0.58
0.6964400000
0 0.144700000
0.5110%
0.8216, co0s
0.8141g.000 0020
0.8023..00s
0.0118
0.5040 o13s
0.5038
0.4764p.0022
0.07600..oss
0.0766%
RL
0. 0,9087c.00022
0.92294g.0011
0.9190g..000
0.D077
0,491
D. 0,3057amm
D. 0.5404.11755
0. 1.50500123
0.0956u.0002
BS
0.9995..c00
0.9305 oo1s
0.9990..
0.0685
0.5712 mox2
0.6215mmm
0.6971n
0.1797uat0
0.3125u.tce
SALUN
0.9985c.0033
0.92733.0007
7.9247a0055
0.0026
0.4845c.
0.3321aams
0.6230muss7
0.2947ua204
0.1143w.113
SG
0. 0.9926c.0029
0.0283
0.5076
0.5037
0.6709
0.0953..1100
0. 0.1037
22


Reviews:
[{"review_id": "rlq6ApCX7M", "reviewer": "Reviewer_dLHn", "summary": "This paper formulates machine unlearning as a Stackelberg Game, where the model owner seeks to unlearn a forget set while preserving the model's utility, and the auditor aims to distinguish the forget set from test data. The authors employ the Implicit Function Theorem and Differentiable Optimization to address the resulting Stackelberg game optimization. Experiments on both image and text datasets demonstrate the effectiveness of the proposed machine unlearning method.", "questions": "1. Could you elaborate on the Membership Inference Attack (MIA) methods utilized within the Stackelberg game framework and the MIA for evaluation metrics? Please provide detailed information.\n\n2. What is the rationale for employing fine-tune-based unlearning in the proposed unlearning framework?\n\n3. The optimization process is not sufficiently clear. Could you please provide more details about Algorithm 1, especially lines 7-9?\n\n4. In Figure 2, please specify the meanings of the x-axis and y-axis coordinates.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The formulation of machine unlearning within a game-theoretic framework is interesting and offers a fresh perspective on the unlearning problem.\n\n2. The paper employs several techniques to enable gradient computation in the Stackelberg game of machine unlearning.", "weaknesses": "1. The experiments conducted exclusively utilize the ResNet-18 model in image tasks, which may restrict the demonstration of the method's applicability across different architectures. Considering more complex models could provide a broader validation of the method's effectiveness and generalizability.\n\n2. The optimization complexity is high, with a computational complexity of $O\\left(n^3\\right)$. It would be beneficial for the paper to explore potential techniques to reduce this complexity.\n\n3. The experimental results on the 20 Newsgroups dataset (Table 1) and in the class-wise forgetting experiments (Table 2) suggest that the proposed unlearning methods may not outperform existing techniques. Further analysis or enhancements might be necessary to establish the advantages of the proposed approach over current methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hr151m7LKz", "reviewer": "Reviewer_HktX", "summary": "This paper introduces a novel adversarial framework for machine unlearning that addresses the challenge of effectively removing data influence from machine learning models without sacrificing performance. The approach leverages a game-theoretic model involving membership inference attacks (MIA) to evaluate the unlearning process. This is innovatively framed as a Stackelberg game, where the unlearner and an auditor (using MIAs) interact in a leader-follower dynamic to enhance unlearning efficacy. The methodology not only emphasizes adversarial unlearning but also integrates advancements in implicit differentiation to streamline the integration of unlearning into existing pipelines.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of a game-theoretic approach to frame the machine unlearning problem is novel and provides a robust theoretical framework to tackle unlearning in an adversarial setting.\n2. The paper successfully integrates complex mathematical tools like implicit differentiation and Stackelberg games, which are sophisticated and not commonly applied in standard unlearning approaches.", "weaknesses": "1. The complexity of the proposed solution, involving advanced mathematical constructs and game-theoretic elements, might pose challenges in terms of practical implementation and computational efficiency.\n2. While the method shows effectiveness in controlled experiments, the scalability of this approach in larger, more heterogeneous datasets and in real-world applications is not thoroughly discussed.\n3. The effectiveness of the unlearning process is heavily dependent on the assumption that the auditor uses specific types of MIAs, which may not generalize well to all potential attack vectors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQAB56DmYh", "reviewer": "Reviewer_Nkr5", "summary": "This paper studies the machine unlearning problem. Specifically, an adversarial machine unlearning method is proposed to maximize the utility of the unlearned model and the efficacy of unlearning. During unlearning, the unlearned model is obtained by minimizing the loss of the retained data and the auditor’s utility through an objective function. The intuition of this method is that if an audit running membership inference attacks cannot distinguish the unlearned data from the testing data, then the unlearned data should be removed from the mode.l Extensive experimental results validate the effectiveness of the proposed method.", "questions": "I have several questions, as outlined in the weakness part.\n\n1, How is the efficacy of the proposed method compared to retraining?\n\n2, What's the performance of the proposed method in unlearning biased samples?\n\n3, Can the proposed method handle sequence unlearning?\n\n4, Will the main conclusion change if measured by state-of-the-art MIAs?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is generally well-written and easy to understand.\n\n+ The proposed method innovatively adopts the idea of adversarial training to solve the machine unlearning problem. Specifically, if MIAs cannot distinguish the unlearned data and the testing data from the unlearned model, then the data can be considered removed from the model.\n\n+ Extensive experiments against various baselines and datasets validate the effectiveness of the proposed method.", "weaknesses": "- Efficacy. Although the authors have some theoretical analysis of the complexity of the proposed method, I am wondering how much faster the proposed method is against retraining. Involving lots of optimization, a direct comparison against retraining would be helpful.\n\n- Unlearning bias samples. What if unlearning samples are not iid from the training data? What’s the performance of the proposed method?\n\n- Sequence unlearning. It seems the proposed method only discusses one-time deletion. Can the proposed method handle sequence unlearn? If so, how would the effectiveness of the proposed method change?\n\n- MIA choices. It seems this paper chooses “simpler” MIAs for the auditor. I am wondering will the main conclusion change, if using the current version of the method, while using the state-of-the-art MIAs [RA] to measure the audit’s utility. Another paper discusses the limitations of weak MIAs [RB]. See as follows.\n\n[RA] Carlini, Nicholas, et al. \"Membership inference attacks from first principles.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n[RB] Hayes, Jamie, et al. \"Inexact unlearning needs more careful evaluations to avoid a false sense of privacy.\" arXiv preprint arXiv:2403.01218 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7FYZ2h3qwt", "reviewer": "Reviewer_N1TT", "summary": "This paper proposes a method of machine unlearning called adversarial machine unlearning.  when an auditor employs MIA to detect the forget data, the unlearner adjusts the unlearned model to make sure the auditor cannot distinguish the forget data and test data. Moreover, this paper uses the Implicit Function Theorem and Differentiable Optimization (DO) to compute the gradients.", "questions": "1.\tIn this paper, the chain rule is applied in Equation (4). For the second term, why is there no term of the form $\\frac{\\partial M(\\tilde{D}_{\\theta_u}^{val}; \\theta_a)}{\\partial \\tilde{D}\\_{\\theta_u}^{val}} \\cdot  \\frac{\\partial \\tilde{D}\\_{\\theta_u}^{val}}{\\partial \\theta_u}\\$?\n \nAs I see, if $\\theta_u$ changes, $\\tilde{D}_{\\theta_u}^{val}$ also changes.\n\n2.\tFor the second term (that has three parts) in Eq.(4), this paper uses two approximations for the first and second part. I have some concerns about the accuracy of using two approximations together. Moreover, can you explain how to compute the third part $\\frac{\\partial\\tilde{D}_{\\theta_u}^{tr}}{\\partial\\theta_u}$?\n3.\tIf there is an error in the Eq.(6)? I think $\\frac{\\partial\\theta_a}{\\partial\\tilde{D}\\_{\\theta_u}^{tr}}=-\\left(\\frac{\\partial f}{\\partial\\theta_a}\\right)^{-1}\\frac{\\partial f}{\\partial\\tilde{D}_{\\theta_u}^{tr}}$ may be the correct one.\n4.\tIt would be better to add some experiments about computation time compared to other unlearning baselines on large datasets.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe presentation of the paper is clear, and the proposed method is easy to follow.\n2.\tDifferent kinds of experiment results are done to show the effectiveness of the proposed methods.", "weaknesses": "1.\tI found another paper [1] that also gives an unlearning algorithm from an adversarial perspective. I think it is important to give a comparison about [1] in the paper, to claim the novelty of this paper. \n2.\tThis method has a time complexity of $O(n^3)$, which is much larger than some unlearning methods. So, I doubt that this method is not applicable in the real world.\n3.\tThere are some concerns that I list in the Questions section about optimization methods. \n4.\tIn Table 1, some results on 20 NewsGroup are not highlighted.\n\n[1] Sharma, Rohan, et al. \"Discriminative Adversarial Unlearning.\" arXiv preprint arXiv:2402.06864 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)