Corrective Machine Unlearning
Authors: Shashwat Goel
Ameya Prabhu
Philip Torr
Ponnurangam Kumaraguru
Amartya Sanyal
Venue: Accepted by TMLR ()

Abstract: Machine Learning models increasingly face data integrity challenges due to the use of large-scale training datasets drawn from the Internet. We study what model developers can do if they detect that some data was manipulated or incorrect. Such manipulated data can cause adverse effects including vulnerability to backdoored samples, systemic biases, and reduced accuracy on certain input domains. Realistically, all manipulated training samples cannot be identified, and only a small, representative subset of the affected data can be flagged.

 We formalize ``Corrective Machine Unlearning'' as the problem of mitigating the impact of data affected by unknown manipulations on a trained model, only having identified a subset of the corrupted data. We demonstrate that the problem of corrective unlearning has significantly different requirements from traditional privacy-oriented unlearning. We find most existing unlearning methods, including retraining-from-scratch without the deletion set, require most of the manipulated data to be identified for effective corrective unlearning. However, one approach, Selective Synaptic Dampening, achieves limited success, unlearning adverse effects with just a small portion of the manipulated samples in our setting, which shows encouraging signs for future progress. We hope our work spurs research towards developing better methods for corrective unlearning and offers practitioners a new strategy to handle data integrity challenges arising from web-scale training.

Published in Transactions on Machine Learning Research (10/2024)
Corrective Machine Unlearning
Shashwat Goel*
strehruainoon@@malllcom
III Hyderabad
ELLIS Institute Tübingen Max Planck Instätute for Intelligent Systems
Ameya Prabhu"
University of Oxford Tübiingen AI Center
Philip Torr Universily of Oxford
Ponnurangam Kumaraguru IIIT Hyderabad
Amartya Sanyal
University of Copenhagen
amsa@di ku, dk
Reviewed on OpenReview: https: openneeviee. net/ formm? id= uBenu4
Abstract
Machine Learning models increasingly face data integrity challenges due to the use of large- scale training datasets drawn from the Internet. We study what model developers can do they detect that some data was manipulated or incorrect. Such manipulated data can cause adverse effects including vulnerability to backdoored samples, systemic biases, and reduced accuracy certain input domains. Realistically, all manipulated training samples cannot be identified, and only small, representative subset of the affected data can be flagged.
We formalize "Corrective Machine Unlearning" as the problem of mitigating the impact of data affected by unknown manipulations on a trained model, only having identified subset of the corrupted data. We demonstrate that the problem of corrective unlearning has significantly different requirements from traditional privacy-oriente unlearning. We find most existing unlearning methods, including retraining-from-sceaa
without the deletion set, require most of the manipulated data to be identified for effective corrective unlearning. However, one approach, Selective Synaptic Dampening, achieves limited success, unlearning adverse effects with just small portion of the manipulated samples in our setting, which shows encouraging signs for future progress. We hope our work spurs rescarch towards developing better methods for corrective unlearning and offers practitioners a new strategy to handle data integrity challenges arising from web-scale training.
1 Introduction
Machine Learning models are increasingly trained on large and diverse datasets, with contributions from numerous users and organizations across millions of web-pages (Schuhmann al., 2022; Gao et al., 2020). However, data integrity issues significantly impact model performance by introducing systemic biases (Prabhu & Birhane, 2021; Konstantinov & Lampert, 2022) and vulnerabilities (Barreno al., 2006b; Sanyal al., 2021; Paleka & Sanyal, 2023). For instance, Carlini et al. (2023) showed that small manipulated subset of web data can lead to large-scale model poisoning, showing the vulnerability of models to adversarial tactics.
"Equal contribution

---

Published in Transactions on Machine Learning Research (10/2024)
Removing identified data and retraining
Legand
Manipulated domain data
Adversery manipulates data
Idantifiid
Unaffected domain data
Developers
Original Model
Ideal Corrective Unlearning
Accuracy using Clean Labels
Figure 1: Traditionally, retraining after removing deletion data is considered a gold standard in unlearning, as all samples whose influence is to be removed are assumed to be known. This relies on the retained data not reinforcing the effect to be unlearnt. When developers cannot. identify all the manipulated data for corrective unlearning, the retained data can continue to perpetuate the adverse effects of the manipulation. Ideally, corrective unlcarning procodures should correct model outputs on the affected domain with access to only a small but. representative subset. of the manipulated data.
A critical real-world obstacle to eliminating this issue is that model developers can only hope to identify a fraction of this manipulated data duc to the size of such largo-scale datascts. One practical way for model developers approach this problem is by using methods for monitoring the integrity of data pipelines (Breck et al., 2019; Wang et al., 2019; Northeutt et al., 2021b) on randomly selected subsets of the whole data. This will identify small, representative subset of the corrupted samples. Having identified this small subset, the primary goal is to "correct" " the negative impacts of corrupted data and their detrimental effect on the model. Due to the high costs already incurred in training and the need to innt uninterrupted service, model developers may wish to update models to remove the influence of the corrupted data instead of abruptly stopping the model's use. We term this problem of removing the influence of manipulated data from a trained model having identified only a small fraction of it, Corrective Machine Unlearning. The goal of Corrective Machine Unlearning is to efficiently eliminate the detrimental effects of the corrupted samples even when the precisc nature and extent of the manipulation is unknown. In this work, the manipulation is characterised by small representative subset of the corrupled samples which created the negalive impact. Since the manipulation type is assumed to be unknown, this implies that corrective unlearning methods should work across different manipulation types.
Our rentral claim is that Corrective Unlearning has different underlying requirements from prior work on unlearning motivated by privacy, and thus needs separate attention. Traditional unlearning algorithms (see Nguyen et al. (2022) for survey) are motivated by the need to cater to user data delction requests in light of privacy regulations (Council of European Union, 2018; California State Leglisature, 2018: Parliament of Canada, 2018). In contrast, Corrective Unlearning algorithms do not need to obtain privacy guarantees on the "unlearned" data. Instead, corrective unlearning algorithms must "correct" the effect of the corrupted training data while only having identified a small subset of said data. We illustrate in Figure why the traditional gold standard the privacy-oriented unlearning literature not sufficient for corrective unlearning.
In addition to highlighting the differences of the corrective unlearning setting from traditional unlearning literature, we benchmark popular unlearning procedures (Kurmanji et al., 2023; Goel et al., 2023; Chundawat et al., 2023b; Foster et al., 2023) in this setting. While there are myriad possible ways of constructing manipulations, we choose two manipulations that represent distinct kinds of real-world threats. First, we study a classic poisoning atlack (Gu et al., 2019), where an adversary can manipulate both features and labels, making the model misclassify samples that contain a specific trigger pattern, hard for model developers notice. Sccond, we study a label-only manipulation called the Interclass Confusion test (Goel et al., 2023), where the adversary incorrectly labels samples between two classes, thereby entangling the model's representations. Such mislabeling can cause systemic biases in model outputs (Prabhu & Birhane, 2021).
2

---

Published in Transactions on Machine Learning Research (10/2024)
Despite being simple manipulations, we find that no unlearning method that we benchmark is able to perform successfully in both scenarios simultaneously. Retraining without identified Deletion data (referred to as RewoD in this paper) is "Exact Unlearning" when all samples whose influence is to be removed are known. Thus it is considered an inefficient gold standard in traditional unlearning literature, so most popular mothods aim to efficiently approximate (Kurmanji et al., 2023; Goel et al., 2023). However, our experiments show that even knowing 80% of the manipulated data not enough for RewoD, and consequently other unlearning methods, to remove the adverse effects introduced by manipulating just 1% of the whole training data. Despite this, we find evidence that corrective unlearning with partial manipulation sets known is possible, as the Selective Synaptic Dampening (SSD) (Foster et al., 2023) algorithm is able to remove the effect of BadNet poisoning with just 10% of the manipulated data being identified. There is much scope for progress though, as SSD fails completely in the Interclass Confusion setting, and also leads lo a significant drop in overall test accuracy when unlearning poisoned data. We hope our work highlights the need for more effective and rigorous study of unlcarning procodures tailored to removing the influence of manipulated data.
2 Problem Formulation
In this section, we formalize the requirements of corrective unlearning, highlight the primary challenges, and detail key differences from traditional privacy-oriented unlearning.
2.1 Motivation
We first motivate the setting from the adversary's and the model developer's perspectives.
Adversary's 's Perspective: The adversary can manipulate any portion of the input data, including labels in supervised learning scenarios. For example, in poisoning attacks, trigger is inserted into each manipulated data sample, altering its label to an incorrect one (Han et al., 2022). The goal of the adversary is to inject certain harmful properties into the model learned using this data. While a poisoning adversary injects the backdoor trigger with malicious intent, another adversary may cause systematic mislabelings, resulting in a biased model. We aim to address both types of adversaries.
Developer's Perspective: Model developers identify some of the compromised data sources after having already trained a model, either through internal monitoring, defenses, or external information like tip-offs. Since the adversary can apply arbitrary manipulations, the exact manipulation type is unknown to the model developer apriori. As such, the corrupted data characterizes the manipulation performed by the adversary. Other ways of characterization, which we are currently out-of-scope for this work include using feedback from model users such incorrect predictions after deployment or threats identified via red-teaming.
While detecting all manipulated data is challenging, removing its effect given small subset may be feasible, if the subset is representative of the broader manipulated set. This often occurs in practice when model developers perform expensive but rigorous monitoring on uniformly sub-sampled subset of the larger dataset. Since this subset is uniformly subsampled, the identified corrupt samples are also a uniformly subsampled subsct of the entire set of corrupted samples. The goal of model developers is to remove the adverse effects of the manipulated data from the original model using this small identified representative subset. We formalize this in the next subsection.
2.2 Objective of Corrective Unlearning
Let X' be the data domain, Y the label space, and P the distribution on X' x Y. Let Sir X y be the training data, and Sm C Ser the training samples manipulated by the adversary, either by modifying features, associated labels, or both. Let Dm C & be the domain where performance is adversely affected when learning using Sm. For example, in poisoning, Dm, contains samples with the poison trigger. In Interclass Confusion, Dm consists of samples from the two affected classes. Clearly, Dm also contains Sm- Finally, let. A be the learning algorithm, and M, = A(S,)) be the trained model.
A corrective unlearning algorithm Ucorr "corrects" the original model Mo by removing the influence of Sm- As mentioned before, we expect only a subset of samples to be identified manipulated, which we denote
3

---

Published in Transactions on Machine Learning Research (10/2024)
the deletion set S Sm. Thus, Ucorr takes as imputs Str, Sy and yields an unlearned model Ma. Let a = |5, /15mm be the fraction of the identified manipulated set used for unlearning. For the rest of this work, we assume that Sy is a uniformly sampled subset of Sm- The performance of a corrective unlearning algorithm is mcasured by how well it optimizes the following two objectives for different values of a. The Taster these two metrics increase as a function of a, the better the performance of the unlearning algorithm.
1. Unlearning:
The primary goal is to unlearn the adverse effect due to the manipulated data Sm. We operationalise this as improving the corrected accuracy on Dm while only knowing an a fraction Sy of the full manipulated set:
Acceor (Ucorr: a) = -((z,))> (x) = y} x E Dml
(1)
where M2 Ucure (Ma, Str. Sy) and 5/1/1s..l-
Utility: An ideal unlearning algorithm should not harm performance on unrelated samples, i.e. outside Dm- Thus, the second goal of the Ucorr to maintain overall accuracy (if not improve it). Using the same nolation as above, we operationalize this as the relain accuracy on X' Dm:
AcCretain (Ucore: a) (( [[[Mu(z) = y | x $ Dm.
Intuitively, the larger the value of a, the higher Acceor should be. When the whole manipulated set is known for deletion i.e. a 1, retraining without it, gives the best outcome. We dub this Uč A(trr \ (m)). The performance of an unlearning algorithm Ceorr can be measured by how large a needs to be for Acceorr (Ucorr: a) to be within some constant fraction (say 0.9) of Acceorr (U 1). We visualize this in our experiments in Section 3.3 and compare different algorithms based on it. Note that, AcCretain is relatively less sensitive to a due to the definition of D... However, it may not always be possible to accurately identify the entire affected domain Dm. Identifying Dm can be particularly difficult when the adversary wants to obscure its true target, as in targeted poisoning attacks (Shafahi et al., 2018), or when it wants to affect the accuracy over the entire domain, such as in indiscriminate poisoning attacks (Barreno et al., 2006a; Biggio et al., 2012). Therefore, formulating these objectives for different settings may require more problem-specific attention. Nevertheless, we set the gold standard for Accretain to also be AcCretain (U 1), and in our experiments, this metric turns out to be largely independent of a.
2.3 Differences from Privacy-Oriented Unlearning
In this section, we discuss the most important distinctions between Corrective Unlearning and Privacy-oriented unlearning. Privacy-oriented unlearning seeks to ensure relrain indistinguishabiilly (Sekhari et al., 2021) )): the unlearning algorithm Ucorr aims to produce models indistinguishabile from the models produced by the learning algorithm trained on just the retain set S,r SS..
Different Goals (Removal of Incorrect Training Data): The goal of privacy-oriented unlearning is to remove the influence of untampered but sensitive user data whereas the objective of corrective unlearning is to remove the influence of manipulated samples.
Implications: Removing uncorrupted but pricote samples in privacy-oriented unlearning scenarios typically degrades model performance (Golatkar et al., 2020a). This is unavoidable as there is a cost to obtain privacy (Jayaraman & Evans, 2019). Some unlearning procedures even nudge the model output towards a uniform distribution over classes for samples in the forget set (Chundawat et al., 2023b; Li & Ghosh, 2023). However, in corrective unlcarning, removing the inffuence of manípulated samples is expected to improve the model's performance the affected domain D,, (i.e. AcCeom). Intuitively, this may, in turn, improve the learned representations thereby also improving overall accuracy Accratiin The underlying cause is that corrective unlearning aims to unlearn corrupt data whereas the focus of privacy-oriented unlearning is removing confidential but not necessarily corrupt data.
Different Gold-Standards (Retraining without deletion set is not enough): In privacy-oriented unlearning, the data whose influence to be removed from the model is specified by user deletion requests. However, as discussed before, when model developers need identify compromised data, it is unrealistic to
4

---

Published in Transactions on Machine Learning Research (10/2024)
assume all of it will be found. Thus, in corrective unlearning at < 1, the retain set Srr \S will continue to contain manipulated data from S.." SS.
Implications: Retraining from scratch on Str \ S is the gold standard for privacy-oriented unlearming but it is computationally expensive. Therefore, at its core, privacy-oriented unlearning is a computational problem. However, in corrective unlearning when a < 1, as Sir \ Sy continues to have manipulated data, unlearning procedures that solely rely on it (Schelter, 2020; Bourtoule et al., 2021; Graves et al., 2021; Goel al., 2023) perpetuate the adverse effects of the manipulation. This necessitates search for algorithms beyond computationally efficient approximations of retraining from scratch, which ceases to be a gold standard.
Interestingly, this introduces a statistical dimension to the unlearning problem. One possible approach is to use the identified manipulated data to detect other manipulated data. Whether this is possible depends mainly on three parameters: size of the identified manipulated set, dimensionality of the data, and complexity of detecting the manipulated data'. For casy-to-detoct manipulations in low dimensions and for large identifiod sets, this problem should be easy. However, existing lower bounds in learning theory (e.g. see Theorem 3.20 in Schapire & Freund (2012)) show that this may well be impossible for small identified sets and hard-to-detee manipulations in high dimensions, which are precisely the realistic conditions we aim to tackle. For example, we show poisoning experiments where the identified subset has 500 samples for 2072-dimensional data and one of the goals of poisons is to be imperceptible. In short, this naturally leads to the following question: How can we efficiently remove the detrimental impacts of Sm using a representative, albeit smaller, subset Sy?
Different Constraints (No Privacy Requirements): Finally, in the corrective unlearning context, Sj and Sm docs not need to be privatized, sctting it apart from privacy-oriented unlcarning.
Implications: Privacy-oriented unlearning is designed to meet strict privacy standards, necessitating either algorithms with theoretical privacy guarantees (Sekhari et al., 2021; Gupta al., 2021) akin to those provided by differential privacy, or at least strong performance against privacy auditing on the data to be forgotten Sy (Golatkar et al., 2020a). This further leads to drop in accuracy as theoretical privacy guarantees are not always tight and lead to underestimation of actual privacy. Such evaluations are orthogonal to the goal of Corrective Unlearning, thereby necessitating the design of new evaluation strategies.
3 Experiments
In this work, we run experiments with two types of manipulations that can occur in real-world data collection pipelines, and test whether existing methods can unlearn their adverse effects. We first describe these manipulations, and then our dataset, model architecture, and unlearning setup.
3.1 Manipulation Types and Evaluation Benchmarks
As a desiderata for corrective unlearning is dealing with corrupted data, agnostic to manipulation type, a good Uoor should correct a broad class of manipulations, including both we test.
Evaluation of Feature-and--Labll Manipulations: Poisoning. When model developers scrape data from webpages, adversaries can manipulate both the data samples and labels. This leaves models trained on this data vulnerable to backdoor attacks, where the model misbehaves in the presence of trigger features known to the adversary, but unknown to the model developers. Prior work has shown this can be realized in real-world settings such as autonomous driving (Han et al., 2022) and models trained on Wikipedia (Carlini et al., 2023). To model this setting, we use the simple BadNet poisoning attack introduced by Gu et al. (2019). We insert a trigger pattern that makes 0.3% pixels white at bottom-right positions in a subset of n training images, re-labeling each of these images to class zero. Here the affected domain Dm, consists of all samples containing the trigger pattern. We benchmark the ability of unlearning methods to remove the effect. of the backdoor after identifying some of the poisoned training data.
Evaluation of Label-only Manipulations: Interclass Confusion. When model developers outsource annotations for their training data, adversarics can only manipulate labels. Systematic mislabeling between
This is based on classical unfform-cooverggnnce based arguments (Alon et al., 1997)
5

---

Published in Transactions on Machine Learning Research (10/2024)
two classes can also occur naturally due to systemic biases in the labelling process, or misinterpretation in annotation guidelines on how to distinguish the classes. While this setting offers lesser power to the adversary, it can still lead to strong mislabeling attacks. Lingam et al. (2024) show that for random label flipping attacks, restricting it to two classes results in the largest drop of accuracy of the Bayes optimal classifier. We thus use the Interclass Confusion (IC) Lest. (Goel el al., 2023), in which, two classes and B are picked, and half of the samples from both classes are selected, and their label is changed to the other class. Models trained on datasets containing this manipulation are more likely to confuse these classes, i.e. predict samples from A as B and vice-versa. The affected domain Dm consists of all samples from class A and class B. We benchmark the ability of unlearning methods to remove the label confusion after identifying some of the confused data.
3.2 Experimental details
We first use the CIFARIO and CIFAR1OO (Krizhevsky et al., 2009) datasets as standard benchmarks in the unlearning literature (Foster et al., 2023; Kurmanji et al., 2023; Chundawat et al., 2023a). We then report poison unlearning results on PCam (Veeling et al., 2018), a binary classification medical imaging dataset, as a potential application. PCam contains histopathologic scans of lymph node sections, with labels indicating the presence of metastatic tissue. For our experiments, we employ the ResNet-9 (Idelbayev, 2018) model for CIFAR10, and the WideResNet-28xii (Zagoruyko & Komodakis, 2016) model for CIFAR100 and PCam.
In the main paper, we report results for 1% of the training data being manipulated, except for the IC test (which has a weaker adversary than poisoning) on CIFAR-10 where 10% manipulation is required for clear observations. In Appendix B we report results for other manipulation sizes, finding similar unlearning trends even when only 0.2% of the training data is poisoned. In each setting, we vary a, the fraction of the manipulated set identified for delction, from 0.1 to 1.0 at intervals of 0.1. For all results, metrics are computed on the test set containing unseen samples. The mean and standard deviation are reported over 3 seeds. In the interclass confusion evaluation, for CIFAR1O, we confuse the Cat and Dog classes, and for CIFAR100, the maple and oak tree classes, to be consistent with the setup of Goel et al. (2023).
Unlearning Methods. We select several stale-of-the-arl unlearning methods to benchmark the effectiveness of current unlearning methods in our setting. Detailed descriptions, along with hyperparameter sweep details for all methods, are provided in Appendix A.2.
(1) Retrain without Deletion set (RewoD): It retrains from scratch on Ser \ Sy using the original training algorithm A. It is considered inefficient but gold-standard oracle in the traditional setting of a 1. Many existing methods are relaxations of this algorithm (He et al., 2021; Graves et al., 2021; Goel et al., 2023).
(2) Catastrophic Forgetting (CF): Goel et al. (2023) show that fine-tuning just the final layers of a deep model on Ser \ S can unlearn label manipulations. We use the strongest version, i.c., fino-tuning all layers.
(3) Selective Synoplic Dumpening (SSD): Foster et al. (2023) selectively dampen learned weights with high influence from Sy, identitied by approximating the Fisher Information Matrix (Martens, 2020). Specifically, they compute cach parameter's relative importance: the ratio of the forget set and retain set loss sensitivity to the parameter. Parameters with relative importance above chosen threshold have their value divided by quantity proportional to the relative importance.
(4) Knowledge Distillation from a Bad Teacher (BadT): Chundawat et al. (2023b) propose making outputs on Sy random by distilling from a randomly initialized network, while distilling the original model on Ser \Sy.
(5) Scalable Remembering and Unlearning Bound (Scrub): Kurmanji et al. (2023) propose a method that alternates between two key steps: (1) distillation away from the original model on Sy to remove the influence of the manipulated data, and (2) knowledge preservation using distillation towards the original model combined with a task-specific loss on Str \ (S to retain useful information.
How to Select Best Hyperparameters for Unlearning? We find choosing the surprisingly tricky question for corrective unlearning. Selecting the model with the best overall accuracy on a validation set (Accretain) may result in low corrected accuracy (Acccor) on the domain affected by the manipulation (Dmx), especially if it is a small fraction of the overall domain X. Moreover, directly measuring the corrective accuracy may not be possible when the correct labels for the deletion set are not known. We
a
6

---

Published in Transactions on Machine Learning Research (10/2024)
Unlearning Method
CIFAR1O
CIFAR100
None
BadT
CF
RewoD
SSD
Scrub
Poisoning
Poisoning
Interclass Confusion
None
90.52
73.79
1.0
BadT
0.29
0.01
0.02 + 0.00
0.8
CF
0.97
0.06
0.40 :: 0.05
CIFAR-10
Acc goff
0.8
0.7
SSD
Scrub
-8.94 + 1.02
0.69 0.06
0.09 + 0.06
-3.25 = 0.53
0.6
RewoD
0.75 0.06
0.23 + 0.07
0.6
Interclass Confusion
0.4
0.5
None
92.36
74.35
0.2
0.4
BadT
0.23 + 0.10
-0.04 = 0.02
CF
0.79 + 0.14
0.31 + 0.07
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Identified Fraction
Identified Fraction
SSD
-1.63 + 0.95
-5.43 = 3.62
Scrub
-3.27 + 1.65
-0.06 = 0.05
0.8
RewoD
0.44 + 0.15
-0.01 = 0.07
0.8
Table 1: Change in retain ac- curacy (AcCretain) after ap- plying different unlearning methods, where higher val- ues are better. Results are re- ported as mean + sidev over 10 values of the identified fractions a from 0 to 1.0. SSD leads to significant drops in model utility.
CIFAR-100
Acc.om
cor
0.6
0.6
0.5
0.4
0.3
0.0
0.2
0.4
0.6
0.3
1.0
0.2
0.2
0.4
0.6
0.8
1.0
Identified Fraction a
Identified Fraction
Figure 2: Corrective Accuracy (Acceorr) after applying different unlearning procedures. Each method ("None" represents the original model) is shown across different identified fractions (a) of manipulated samples. No method unlearns both the manipulations well when 80% of the manipulated data is identified, including RewoD which is considered gold standard at a =1
propose to use weighted average between overall accuracy (utility) on validation set and the fraction of Sy samples where the model's classification changes (unlearning). We weigh them equally for our results.
3.3 Experimental Results
Figure 2 shows corrective accuracy trends that quantify the efficacy of different methods al unlearning the adverse effects of manipulations. RewoD is the gold standard when all manipulated samples are known, and indeed it shows the highest accuracy when |Syl |Smll For the IC evaluation, RewoD demonstrates consistent improvement as more deletion set samples shows no improvements when developers detect less than 80% of the manipulated samples, even where only 1% (500 samples) of the training data is manipulated. This highlights the insufficiency of the privacy-based unlearning goal approximating retraining from scratch on S,, ASy, as the remaining poisoned samples can maintain their adverse effects, even when their number small (Gu al., 2019). As a consequence, popular approaches in unlcarning litcrature like Scrub and CF do not perform well the poisoning sctting. Scrub performs better than CF in the poisoning setting, but the opposite is true in the interclass confusion setting. BadT shows poor results in both settings, randomizing outputs Sy conflicts with the goal of correcting the model on the affected domain.
identified. However, for the poisoning evaluation, it
On the contrary, SSD recovers Accrom for poisoning even after identifying just 10% of the manipulated samples, demonstrating manipulations can be unlearnt with a small fraction of manipulated samples identified. However, SSD completely fails for the IC test, providing no improvements over the original model. Further, as shown in Table 1, SSD leads to significant drops in model utility (Accretaiu), while other unlearning methods maintain utility. Note that report averaged Accretain across
find it to largely be independent of a.
7

---

Published in Transactions on Machine Learning Research (10/2024)
Conclusion: Traditional unlearning methods that train Str \ S perform poorly in practical scenarios when not all manipulated samples are known. SSD shows positive results for removing poisons, demonstrating corrective unlearning is possible at a < 1, though it fails completely at removing interclass confusion and hurts model utility, leaving scope for improvements.
3.4 Why does SSD fail for IC test but work for poisoning?
SSD selects which parameters to dampen using the ratio of the parameter's importance for the forget set versus the retain set. We hypothesise that for poisons, a few parameters are disproportionately more important for the forget set than others, whereas for the IC test, the effect is more distributod among the parameters and no small subset of parameters particularly important. If this is the case, an approach that dampens high- influence parameters is likely to target a "relevant" parameter in the poisoning setting, effectively reducing its adverse effects, but fail to remove differentially interclass confusion without hurting retain accuracy. To test this hypothesis, we show the distribution of the importance ratio (used to select the weights dampen) for the two tests in Figure For poisoning, we observe few outlier weights with disproportionately high values (e.g., 800), whereas for the IC test, the distribution does not show such extreme values (the maximum is less than 200). This empirical observation confirms our hypothesis.
10"
Poison
of of
10
10"
10
10"
10'
10
200
400
600
800
Importance Ratio
Figure 3: Histogram of importance ra- tio values computed for each parame- ter by SSD. We find poisoning leads to more outlier values, which supports the hy- pothesis that poisoning can be removed by dampening outlier parameters unlike Inter- class Confusion.
SSD succeeds at removing poisons because can dampen weights which correlate the poison feature with the wrong label, even with small portion of the manipulation set known. This is consistent with the well-kmown strategy of pruning small subset of weights to mitigate poisons (Wang et al., 2019), as they correlate specific feature with the mislabel. This may motivate the tractability of mechanistic interpretability- based approaches for unlearning certain types of poisoned data (Elhage al., 2021).
Undeaming Metuol
EadT
RewD
Scoe
0.8
Acc cont
0.7
0.6
3.5 Application on a Medical Dataset: PCam Binary Classification
0.5
We now examine if our findings generalize to PCam (Veeling et al., 2018), an application-specifie dataset used in medical imaging. The only change we expect is that in binary classifi- cation tasks, knowing which class is incorrect fully specifies the correct class. Therefore, methods like Scrub which do form of gradient ascent, moving away from the original manipulated label, are expected to perform well. In Figure 4, we observe that RewoD, CF, and BadT contimue to perform poorly until most of the manipulated data is detected, while SSD performs well even al small a. As anticipated, the only difference that Scrub shows significant improvements as moving away from the original model's outputs the forget set Sy infers the correct labels.
0.7
0.4
0.6
0.8
1.0
Identified Fraction a
Figure 4: Corrective Accuracy (Acccore) for unlearning poisons across different methods on PCam. Unlearning methods that move away from the original label, such as Scrub, perform well even when less ma- nipulated data is detected, as in this case there is only one other possible label.
4 Related Work
Learning from manipulated Data: The adverse effects of manipulated training data on machine learning models well-documented across objectives like fairness (Konstantinov & Lamperr, 2022), robustness
8

---

Published in Transactions on Machine Learning Research (10/2024)
(Sanyal et al., 2021), and adversarial reliability (Paleka & Sanyal, 2023; Tian et al., 2022). One line of defense is designing training strategies more robust to these issues, see Song et al. (2022) for a survey on learning with mislabels. However, learning robust models from manipulated data is a hard problem as reduced sensitivity to such minority data populations can harm accuracy and fairness (Sanyal et al., 2022). Unlearning specific samples which are discovered to be manipulated can be a complementary post-training mitigation approach.
How to detect manipulated data? A curious reader would wonder if the critical task is detecting the small subset of the data itself. However, detecting manipulated data has long been studied (Brodley & Friedl, 1999), with prior work detailing techniques to discover mislabeled (Pleiss et al., 2020; Northcutt et al., 2021a), biased (Prabhu & Birhane, 2021; Jiang & Nachum, 2020) and poisoned (Chen et al., 2019; Wang et al., 2019) data. We assume the model developers employ such strategies for monitoring their data sources. However, they cannot simply throw away the trained model when manipulated data is found due to expensive retraining costs. Our goal to study how to cheaply mitigate adverse effects on such models using unlearning.
Known manipulations: If the type of manipulation is known, one may employ maniipuaation-specife mitigation techniques such as poisoning defences against data poisoning attacks (see Goldblum et al. (2022)
for survey) or erasing concepts like artistic style from visual models (Gandikota et al., 2023). If the samples can be corrected through re-annotation, one may also use knowledge editing techniques (Mitchell et al., 2022). We restrict the scope of our work to not knowing the precise manipulation, as corrected data often cannot. be obtained, we hope to use unlearning as a broader, panacea procedure across known and unknown data manipulations.
Most prior work in designing unlearning procedures is motivated by privacy applications, and aims to achieve retrain indistinguishability (Ginart al., 2019; Golatkar et al., 2020a), that is to create distribution of unlearnt models indistinguishable from retraining from scratch without the data to be deleted. In Section 2.3 we discuss differences in corrective unlearning desiderata from retrain indistinguishaabliiy, "Exact Unlearning" procedures ensure the unlearnt model never sees the data whose influence is to be deleted by design of the training procedure (Bourtoule et al., 2021; Schelter, 2020). The empirical results of RewoD in Section 3.3 show how these approaches may not sulfice for corrective unlearning when the full manipulation set is unknown. Moreover, such methods drastically deteriorate in efficiency as the number of samples to delete increase (Warnecke et al., 2021). This has led to "Inexact Unlcarning" proposals, and we discuss different paradigms in image classification in Appendix A.2, and use state of the art methods from each paradigm in our experiments. Prior work has also shown that techniques like sparsification (Jia et al., 2023) and regularization (Thudi et al., 2022a) of the original model can improve the performance of unlearning methods. A group of works (Izzo et al., 2021; Wu et al., 2020; Gupta et al., 2021; Neel et al., 2021; Thudi et al., 2022b; Sekhari et al., 2021) also study unlearning procedures on convex or linear models with theoretical guarantees inspired from differential privacy (Dwork, 2006), but in this work we focus on deep models.
Unlearning:
Some unlearning works have separately evaluated on both removing backdoors (Sommer et al., 2022; Wei et al., 2023; Liu et al., 2022) and mislabeled samples (Kurmanji et al., 2023; Goel et al., 2023). Indeed, our work proposes neither new methods, nor new evaluations. Instcad, we unify these two types of manipulations, to characterize the corrective unlearning as being fundamentally different from the privacy-oriented setting. Kurmanji et al. (2023) suggested different applications of unlearning require different methods. We build on this by formulating the requirements of corrective unlearning, crucially finding retraining cannot be usod as a gold standard as all deletion data may not be known, adding a statistical dimension beyond unlearning being just computational problem. Finally, some prior work has explored unlearning using few even zero samples, but they either require generative models to unlearn classes (Yoon et al., 2023; Chundawat et al., 2023a) or "counterfactual samples" for unlcarning biases (Chen et al., 2024b;a), which is non-trivial to apply for corrective unlearning of manipulations.
5 Limitations, Broader Impact, and Conclusion
Limitations and Future work Given the failure of state-of-the-art unlearning methods (including RewoD) at correcting our two simple manipulations, we didn't include more complex manipulations. Ideal corrective unlearning approaches should exhibit robustness against a broad spectrum of manipulation types. Specifically, these methods should withstand adaptive attacks, where the manipulations targeted for unlearning are
9

---

Published in Transactions on Machine Learning Research (10/2024)
crafted with knowledge of the unlearning procedures themselves (Tramer et al., 2020), not just the two manipulations we study. We hope future work will design adaptive adversaries and corresponding unlearning algorithms that can correct manipulations introduced by these adversaries. This provides scope to design stronger evaluation frameworks for corrective unlcarning. Apart from manipulating features and labels, adversaries could generate entirely synthetic samples (Zhang et al., 2019; Huang el al., 2020). Although our focus is on supervised image classification, the concept of manipulation and its correction is also relevant in self-supervised learning contexts, such as language modeling (Wallace al., 2021). An additional complexity could be the presence of false positives, where clean sample gets identified as manipulated. Corrective unlearning may not always be tractable in all of these settings. Thus, apart from designing new algorithms, we believe Corrective unlearning can inspire a new line of theoretical works, including fundamental questions such as: what are necessary and/or sufficient. conditions and size for a 'representative set' of manipulated samples so that corrective unlearning is possible? Can we design procedures to infer more manipulated samples from a representative subset? Can we design algorithms with unlearning guarantees on Acc ACCretain instead of indistinguushabilitty of distributions? We hope these questions guide future work on corrective unlearning.
Broader Impact Corrective unlearning is post-training strategy designed to mitigate the effects of manipulated incorrect training data. The societal benefits of this approach are similar to those achieved by defending against manipulations that introdluee backdoors or systematic biases, which are largely positive. In some scenarios, such as collective action (Hardt et al., 2023) against harmful models, the ability to manipulate training data could be beneficial. However, defenses against such manipulations might have negative socictal effects. These ideas remain speculative, and overall, we believe that removing the effects of manipulated data will have a positive societal impact.
Conclusion Overall, we characterize the Corrective Machine Unlearning setting, designed to mitigate the negative effects of manipulated data discovered post-training, such diminished accuracy on specific parts of the domain. In contrast to the assumption of fully specified user deletion requests in prior work on unlearning, we acknowledge that all the manipulated data samples may not be known. Developers can use a small representative subset of the manipulated samples, which can be collected by careful inspection of a uniform subsample of the training data. We discuss how this leads to significant changes from the traditional unlearning setting, which primarily designed to address privacy concerns.
Our findings indicate that popular unlcarning methods, even retraining on the remaining data (RewoD), fails to enhance accuracy on manipulated domain samples unless nearly all of the manipulated data is identified. A notable exception is SSD, which shows promise by successfully mitigating the effects of the BadNet poison, thus illustrating the feasibility of counteracting manipulated data with only a partial subset identified. However, this method does not generalize across scttings, such as addressing Interclass Confusion manipulations. We hope our work spurs the development of stronger corrective unlearning methods to help practitioners deal with data quality issues arising from web-scale training.
Acknowledgements
SG was fundod by an Effective Ventures Foundation (UK) grant as part of the ML Alignment Theory Scholars (MATS) program. AP was funded by Meta AI Grant No. DFR05540. PT thanks the Royal Academy of Engineering and FiveAT for their support. This work is supported in part by UKRI grant: Turing AT Fellowship EP/W// /W002981/1 and EPSRC/MURI grant: EP/N019474/1. The authors would like to thank (in alphabetical order): Arvindh Arun, Shyamgopal Karthik, Shashwat Singh, Shiven Sinha, Vishaal Udandarao, Christian Schroeder de Witt for helpful feedback, and Varshita Kolipaka for contributing illustrations.
10

---

Published in Transactions on Machine Learning Research (10/2024)
References
Noga Alon, Shai Ben-David, Nicolo Cesa-Bianchi, and David Haussler. Scale-sensitive dimensions, uniform convergence, and learnability. Journal of the ACM (JACM), 1997.
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and Doug Tygar. Can machine learning be secure? In Proceedings of the 2006 ACM Symposium Information, computer and communications securily, 2006a.
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug Tygar. Can machine learning be secure? In ACM Symposium on Information, Computer and Communications Security (ASIA CCS), 2006b.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning atlacks against support vector machines. In International Conference on Machine Learning (ICML), 2012.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Shoquette-Choo Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlcarning. In IEEE Symposium on Security and Privacy (IEEE S&P), 2021.
Eric Breck, Marty Zinkevich, Ncoklis Polyzotis, Steven Whang, and Sudip Roy. Data validation for machine learning. In Proceedings of SysML, 2019.
Carla E Brodley and Mark A Friedl. Identifying mislabeled training data. Journal of Artificial Intelligence Research (JAIR), 1999.
California State Leglisature. California consumer privacy act. 2018.
http:p//weer......ggggaaa.teeete httpe//....../cccccaaaeeetteeeee xhtml http::/hhhmii1111111000000000100
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramer. Poisoning web-scale training datasets is practical. arXiv preprint ar arXi::2003. 02444, 2023.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Ruizhe Chen, Jianfei Yang, Huimin Xiong, Jianhong Bai, Tianxiang Hu, Jin Hao, Yang Feng, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. Fast model debias with machine unlearning. Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Ziheng Chen, Jia Wang, Jun Zhuang, Abbavaram Gowtham Reddy, Fabrizio Silvestri, Jin Huang, Kaushiki Nag, Kun Kuang, Xin Ning, and Gabriele Tolomei. Debiasing machine unlearning with counterfactual examples. aiviv preprint orXiv:2404.15700 2024b.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 2023a.
Vikram Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI Conference on Artificial Intelligence (AAAI), 2023b.
Council of European Union. Eu general data protection regulation. 2018. https://eur-1es lex. auropa.eu//egeee content//E///X///P////rrttt 3200
Cynthia Dwork. Differential privacy. In International Colloguium on Automata, Languages and Program- ming (ICALP), 2006.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds; Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Trunsformer Circuils Thread, 2021. https://taansfrmerrcrrriiiis
/index. ml.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. arXiv preprint ar arXi:23333.02798 2023.
Robert M. French. Catastrophic forgetting in connectionist networks. In Trends in Cognitive Sciences, 1999.
Rohit. Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 2426-2436, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiu, 2020.
Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making Al forget you: Data deletion in machine learning. In Advances in Newral Information Processing Systems (NeurIPS), 2019.
Shashwat Goel, Ameya Prablu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXie preprint arXi::2306.1100 2023.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In TEEE Conference on Computer Vision and Pattern Recognation (CVPR), 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box:
Scrubbing deep networks of information accessible from input-output observations. In European Conference Compuler Vision (ECCV), 2020b.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring al altks on deep neural networks. IEEE Access, 2019.
Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharif--Mavvajedii, and Chris Waites. Adaptive machine unlearning. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
Xingshuo Han, Guowen Xu, Yuan Zhou, Xuchuan Yang, Jiwei Li, and Tianwei Zhang. Physical backdoor attacks to lane detection systems in autonomous driving, In Proceedings of the 30th ACM International Conference on Mullimedia, 2022.
Moritz Hardt, Eric Mazumdarr, Celestine Mendler-Dünner and Tijana Zrnie. Algorithmic collective action in machine learning. In International Conference on Machine Learning (TCMT). PMIR, 2023.
Yingzhe He, Guozhu Meng, Kai Chen, Jinwen He, and Xingbo Hu. Deepobliviate: A powerful charm for crasing data residual memory in deep neural networks. arXiv preprint, 2021.
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general- purpose clean-label data poisoning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Yerlan Idelbayev. Proper ResNet implementation for CIFARIO/CIAARIM in PyTorch. https: // gitbb com/ wwwmaster/ pytorch_ resnet_ cifar10, 2018.
Zachary Imzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
Bargav Jayaraman and David Evans. Evaluating differentially private machinc learning in practice. In 28th USENIX Securily Symposium (USENIX Securily), pp. 1895 1912, 2019.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Advances in Neural Information Processing Systems (NewrIPS), 2023.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Nikola H Konstantinov and Christoph Lampert. Fairness-aware pac learning from corrupted data. Journal of Machine Learning Research (JMLR), 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's ss Thesis, Department of Compuler Science, University of Toronlo, 2009.
Meghdad Kurmanji, Peter Triantafillou, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Junde Li and Swaroop Ghosh. Random relabeling for efficient machine unlearning, arXiv preprint arXiw:2303.0123, 2023.
Vijay Lingam, Mohammad Sadegh Akhondzadeh, and Aleksandar Bojchevski. Rethinking label poisoning for GNNs: Pitfalls and attacks. In International Conference Learning Representations (ICLR), 2024. URL
https:// net/forummidd-Jiieeeee
Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022. IEEE Conference on Computer Communications (IN- FOCOM), pp. 280-289. IEEE, 2022.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research (JMLR), 2020.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning (ICML), 2022.
Seth Neel, Aaron Roth, and Saced Sharifi- Maaaifii-Mavjerdi.. Descent-to-delete Gradient-basce methods for machine unlearning. In Conference on Learning Theory (COLT), 2021.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chune Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXio preprint arXiv:2209.0220 02299., 2022.
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Advances in Newral Information Processing Systems (NeurIPS), 2021a.
Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty dataset labels. In Journal of Artificial Intelligence Research (JAIR), 2021b.
Daniel Palcka and Amartya Sanyal. A law of adversarial risk, interpolation, and label noise. In International Conference on Learning Representations (ICLR), 2023.
Parliament of Canada. Personal information protection and electronic documents act. 2018. https:/ h/www//wwwwwwww http://wwwwp......////////. priv. gg. http////opcceww///eeeee /
13

---

Published in Transactions on Machine Learning Research (10/2024)
Alexandra Peste, Dan Alistarh, and Christoph H Lampert. Ssse:
Efficiently crasing samples from trained machine learning models. In NewrIPS Workshop on Privacy in Machine Learning (NeurIPS- W), 2021.
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Newral Information Processing Systems (NeurIPS), 2020.
Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? In IEEE
Winter Conference on Applications of Computer Vision (WACV), 2021.
Amartya Sanyal, Puncet K. Dokania, Varun Kanade, and Philip Torr. How benign is benign overfitting? In International Conference on Learning Representations (ICLR), 2021.
Amartya Sanyal, Yaxi Hu, and Fanny Yang. How unfair is private learning? In Conference on Uncertainly in Artificial Intelligence (UAI), 2022.
Robert E Schapire and Yoav Freund. Foundations of machine learning. 2012.
Sebastian Schelter. 'amnesia' machine learning models that can forget user data very fast. In Conference on Innovative Data Systems Research (CIDR), 2020.
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
Ayush Sckhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2018.
David M. Sommer, Liwei Song, Sameer Wagh, and Prateek Mittal. Athena: Probabilistic Verification of Machine Unlearning. Proceedings on Privacy Enhancing Technologies, 2022.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In IEEE European Symposium on Security and Privacy (EuroS&P), pp. 303-30 309. IEEE, 2022a.
Anvith Thudi, Hengrui Jia, Tlia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In USENIX Security Symposium (USENIX), 2022b.
Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. A comprehensive survey poisoning attacks and countermeasures in machine learning. ACM Computing Surveys (CSUR), 2022.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer-Assiste Intervention (MICCAT). Springer, 2018.
Eric Wallace, Tony Zhao, Shi Feng, and Samcer Singh. Conccaled data poisoning attacks on nlp mod- cls. In North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACI-HI.T), 2021.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium Security and Privacy (IEEE S&P), 2019.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. Network and Distributed System Security Symposium (NDSS), 2021.
Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Back- door mitigation by unlcarning shared adversarial examples. Advances in Neural Information Processing Systems (NewrIPS), 36:25876-25900 2023.
Yinjun Wu, Edgar Dobriban, and Susan B. Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning (ICML), 2020.
Youngsik Yoon, Jinhwan Nam, Hyojeong Yun, Jaeho Lee, Dongwoo Kim, and Jungseul Ok. Few-shot unlearning by model inversion, 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Mochine Vision Confer- ence (BMVC), 2016.
Jiale Zhang, Junjun Chen, Di Wu, Bing Chen, and Shui Yu. Poisoning attack in federated learning using generative adversarial nets. In IEEE International Conference on Big Data Science and Engineering (Trust- Com/BigDataSE), 2019.
15

---

Published in Transactions on Machine Learning Research (10/2024)
A Experimental Setup Details
A.1 Training Details
Our standard training procedure A is as follows: We train our models for 4000 steps on CIFAR1O, PCAM and 6000 steps on CIFARI00. Each step consists of training on a single batch, and we use a batch size of 512 throughout. We use an SGD optimizer with momentum 0.9 and weight decay 5c-4, a linear scheduler with Imuit = 1.25, and warmup steps as TOO of the total training steps. The same hyperparameters are used during unlearning unless otherwise specified. The setup used for all experiments is a PC with Intel(R) Xeon(R) E5-2640 2.40 GHz CPU, 128GB RAM and 1 GeForce RTX 2080 GPU.
A.2 Detailed Description of Unlearning Methods
To benchmark the performance of existing unlearning proposals on corrective unlearning scenarios, we select the strongest unlcarning methods across five popular paradigms:
(1) Retrain without Deletion set (RewoD): This paradigm involves retraining parts of the ML system (Bourtoule al., 2021; Goel et. al., 2023; He et al., 2021) that are influenced by Sy from scratch using Sey \Sy.
Method Used: We benchmark the strongest version, retraining the entire model from scratch on S.y \ Sy using the original training algorithm A. This is considered an inefficient gold standard in prior work.
(2) Catastrophic Forgetting (CF): Neural Networks suffer from ceasstrophi--forgettii (French, 1999)- when a model is updated without some previously learnt samples, the model loses knowledge about them. Many unlearning methods perform finetuning on S.r \ Sy to achieve unlearning of S via catastrophic forgetting, and Goel et al. (2023) showed weaker efficient version performs well on the IC test.
Method Used: We finetune using the original training procedure A for 1000 steps on Ser SS.
(3) Modifying learnt parameters with high influence from Sy: This is training- free class of methods (Golatkar et al., 2020a;b; Peste et al., 2021; Chundawat et al., 2023a) that identifies parameters with information relevant to the forget set using statistics like the Fisher Information Matrix (FIM), It then damages these parameters by adding noise or reducing their magnitude hoping to selectively remove information about Sy-
We benchmark the recently proposed Selective Synaptic Dampening (SSD) method which has shown state of the art results in this paradigm (Foster el al., 2023). We extensively tune the weight selection threshold and weight dampening constant y. We find when should be tuned relative to a for optimal results. For each datapoint, we pick the best result out of runs with a = [0.1, 1, 10, 50, 100, 500, 1000, 1e4, 1c5, le6), 7 = 10, 0.5a, at 5ax 10a].
Method Used:
(4) Pushing Sy outputs towards random: Some unlearning procedures (Graves et al., 2021; Li & Ghosh, 2023; Chundawat et al., 2023b) push the model towards random outputs on the deletion set.
We benchmark Knowledge Distillation from Bad Teacher (BadT) (Chundawat et al., 2023b), state of the art method in this paradigm, which simultaneously distills from randomly initialized neural network on Sy, and the original model on the remaining data S,y \ ASy. We finetune the original model using this procedure for 1000 unlearning steps.
Method Used:
(5) Alternating between Forgetting and Preservation Steps:
Method Used: Kurmanji et al. (2023) propose SCRUB and show it performs well on unlearning mislabelled samples when all are identified. The method alternates between forget steps and knowledge preservation steps. The forget step involves doing gradient ascent using the task-loss for Sy. The knowledge preservation step does knowledge distillation from M, using S,, SS as well as optimizing the task-loss on S., \ Sy. We finetune the original model using this procedure for 1000 unlearning steps, out of which the forget step is used only in the first. 200 unlearning steps as it is recommended in the paper to it only in the initial
16

---

Published in Transactions on Machine Learning Research (10/2024)
Dataset
#Classes
Model
Polsoning (8w()s..)
IC Test iSuul/ Rcrl
CIFAR-10 10
10
ResNet-0
0.2% 1%, 2%
1%. 5%. 10%
CIFAR-100
100
WideReaNet-28410
0.2%, 1%, 2%
0.2%, 0.5%, 1%
Table 2: Dataset, models and manipulation sizes for the Poisoning and Interclass Confusion (IC) evaluation.
iterations. We use a smaller learning rate (0.0025) as the original value leads to stability issues. We lune the hyperparameter a which controls the trade-off between the distillation loss and the task-loss. For each datapoint, we pick the best result out of runs with a [0.001, 0.01, 0.05, 0.1, 0.5, 1,5, 10].
B Further Results
We now provide results not included in the main paper due to space considerations. Specifically: (1) We report results across different manipulation set sizes (settings summarized in Table 2), on both unseen samples from the affected domain Dm (generalization effect of manipulation) and the manipulation set used in training Sm (memorization of manipulation). We do this for both the poisoning and IC evaluation. (2) We report computational elficiency by measuring unlearning time for each method. (3) We report hyperparameler sensitivity analysis for all methods.
B.1 Unlearning (Acccorr) results across Manipulation Sizes, and on the Manipulated Set Sm
Setup In this section, we vary the manipulation sizes for both poisoning and interclass confusion, to see if the trends are consistent. As the adversary is less powerful (label-only) in IC, we expect 1C will need larger manipulation set sizes to show clear trends, while poisoning will work with very small manipulation sets. Due to the changing manipulation sizes, we report the deletion set size on the X-axis, with the total manipulation set size labeled below each subfigure, and also noticeable as the right-most point on the X-axis. In the main papor, we reported unlearning results (Acccorr) on unscen samples from the affect domain Dm, but here we will also investigate the same metric computed on the manipulated samples used in training itself. We thus use the more general term of "clean-label accuracy", i.e. accuracy computed with respect to correct labels, the Y-axis, which is the same as Accoorr for results unseen samples.
Unlearning Poisoning In Figure 5a, we report results on a smaller (0.2%) and bigger (2%) manipulation size than the main paper (1%, also reported here for reference). In both cases, and notably even for the smaller manipulation set, we find consistent results. To measure the removal of mislabelling on poisoned training samples, we report clean-label accuracy on Sn in Figure 5b. The trends across unlearning methods are similar to the ones unseen samples from the affected domaín Din: though the absolute accuracies after unlearning are higher expected from training samples in comparison to test set samples. The success of SSD in producing an unlearnt model that successfully classifies the manipulated training data shows how an ideal unlearning algorithm can help re-label the detected data correctly.
Unlearning Interclass Confusion In Figure 6a, we report results on smaller manipulation sizes (0.2%, 0.5% for CIFARI00, and 2%, 5% for CIFAR10) than the main paper (same sizes, i.c. 1% on CT- FAR100 and 10% on CIFARIO, also reported here for reference). We see similar trends but with weaker fidelity on the medium setting, but no clear observations While the smallest manipulation size (subfigures a, d) for Interclass Confusion did not show significant effects on unseen samples from class A, B, Figure 6b shows unlearning methods continue to give wrong predictions on the class A, B samples used for training. This emphasises the need to check unlearnt model outputs on unseen and training samples from the affected domain Dm separately, especially when the manipulation set is small to have an effect on model behaviour for unseen samples.
unseen samples in the smallest manipulation size.
B.2 Computational Efficiency
In Table we report average unlcarning times of different unlcarning methods. In the case of RewoD and CF, while more efficient relaxations have been proposed (Goel et al., 2023; He al., 2021; Graves al., 2021), we retrain from scratch to perform the stonggtt unlearning, which we still find to be insufficient.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Unlearning Method
None
BadT
CF
RewoD
SSD
Scrub
1.0
1.0
1.0
CIFAR-11 -10 Clsar-label Accuracy
Clean label Accuracy
Glan-label Accuracy
0.8
0.8
0.8
0.6
0.4
0.4
0.2
0.2
20
103
200
500
200
400
600
800
1000
Deletion Size
Deletion
Deletion Size
(a)
|sm 100(0.2%)
|mml
500(1%)
(c) Sm = 1000(2%)
0.8
0.8
Clcan-labcl Accurocy
Clean- sennlaablll Accuracy
Clean-label Accuracy
CIFAR-100 100
0.5
0.6
0.4-
0.4
0.2
20
40
60
80
100
0.0
100
200
300
400
500
200
800
1005
Deletion Size
Delction Size
Deletion Sizc
(d) |Sml 100(0.2%)
(e) |mm
500(1%)
(f) [Sm 1000(2%)
(a) Clean-label Accuracy Test Samples with Poison Trigger. Each method is shown across deletion sizes
Syl after unlearning ("None" represents original model). Existing unlearning methods except SSD, including RewoD which is traditionally considered a gold-standard, perform poorly (b), (c), (e), (f) when 80% of the poisoned data is identified for unlearning, even when just 1% training data poisoned as in (b), (e).
Unlearning Method
None
BadT
CF
RewoD
SSD
Scrub
1.0
1.0
Clean-ibel Accuracy
Clean-label Accuracy
Clean-labe label Accuracy
0.8
0.8-
CIFAR-10 -10
0.6
0.6
0.6
0.2
0.2
200
400
500
200
400
600
800
1000
Deletion Size
Deletion Size
Deletion Size
(a)
[Smm 100(0.2%)
|.mm|
500(1%)
(c) [mm
1000(2%)
1.0
1.0
Clean-label Accuracy
Clean Accuracy
Cean-llbel Accuracy
0.8
0.6
CIFAR-100
0.8
0.6
0.4
0.4
0.2
0.2
40
60
80
100
100
200
300
0.0
200
400
600
800
1000
Deletion
Deletion Size
Delction Size
(d) |.mml 100(0.2%)
(c) [Sm 500(1%)
(f) |Sm. 1000(2%)
(b) Clean-label Accuracy on Manipulated Train Samples Sn, with Poison Trigger. Each method is shown across deletion sizes |Sy] after training with adversarial poisoning None represents the original model). Trends mimic results for clean-label accuracy on unseen samples with the poison trigger.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Unlearning Method
None
BadT
CF
RewoD
SSD
Scrub
0.95
Clean-label Accuracy
Clcan-laeel Accuracy
0.8
CIFAR- 10
0.8
0.7
Clean-label
0.6
0.80
0.7
0.5
0.75
0.6
0.70
100
260
Deletion Size
300
500
500
1000
Deletion
1500
2000
2500
1000
2000
Delction Sizc
3000
4000
5000
(a) |mm| 500(1%) (1%)
(b) |Sml 2500(59)
(c) [Sm]
5000 10%)
Cleen-label Accuracy
0.85
Clean-label Accuracy
0.85
Clean-labe Accuracy
0.8
CIFAR-115 100
0.80
0.7
0.90
0.75
0.75
0.70
0.60
0.3
0.55
100
150
200
250
0.2
200
Deletion Size
Deletion Size
Deletion
(d) Smm 100(0.2%)
(e) |Sm 250(0.5%)
[ S 500(1%)
(a) Clean-label Accuracy on Test Samples on the Two Confused Classes. We compute clean-label accuracy improvements over original model (represented as "Noue"), and other unlearning methodd also require a large fraction of the manipulated data to be identified for unlearning. In the lower manipulation size setting (a) and (d), the model outputs on unseen samples are not affected much, so we show un unlearning trends on manipulated train samples below.
the classes A, B used for the Interclass Confusion Lest, across deletion sizes |Sy]. SSD provides
Unleaming Method
None
BadT
CF
RewoD
SSD
Scrub
CIFAR- 10 Clean label Accuracy
Clean-label Accuracy
Clean-label Accuracy
0.8
0.6
0.6
0.4-
0.4
0.2
0.2
0.2
100
200
300
400
500
500
1000
1500
2000
2500
0.0
1000
2000
3000
4000
5000
Delction Sizc
Deletion Size
Deletion Size
(a) |.ml 500(19) (1%)
(b) |Sml 2500(5%)
(c) |Sm.l 5000 10%)
1.0
1.0
Clean-label Accuracy
Cleandabell Accuracy
0.8
Clean-label Accuracy
CIFAR-15 CIFAR-100 -00
0.8
0.6
0.6
0.4
0.4
0.2
100
0.0
250
0.0
100
200
300
400
500
Deletion
Deletion Size
(d) [S 100(0.2%)
(e) |Sa,] 250(0.5%)
(f) Sm
500(1%)
(b) Slean-label Accuracy on Manipulated Training Samples S.. with Interclass Confusion for different
unlearning methods ("None" represents original model) across deletion sizes |Syl Existing unlearning methods
perform poorly when 18 is lower. Even the smallest. setting (10% of single class) shows clear unlearning trends.
3m
19

---

Published in Transactions on Machine Learning Research (10/2024)
Method
Time (minutes)
Rewol
49.93
CF
10.52
SCRUB 16.86
SSD 1.80
BadT
33.19
Table 3: Unlearning Time by Method
0.8
0.6
Acterr
AcCouv
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.
Deletion fraction a
Deletion fraction o
Figure 7: Unlearning performance of Scrub across hyperparametors on the IC evalua- tion at different a.
Figure 8: Unlearning performance of Scrub across hyperparameters on the poisoning evaluation at different a.
B.3 Hyperparameter Sensitivity
How sensitive are our claims to hyperparamcters? For CF, RewoD and BadT, there are no crucial hyperpa- rameters apart from the ones used in the original training procedure, for which we continue to use the same ones as we found them to work quite well. To check hyperparameter sensitivity for Scrub, SSD, we analyze two kinds of plots on the CIFAR10 dataset:
Scatter plot of Unlcarning across deletion fraction: We plot the Acc for the model produced by each hyperparameter at values of from 0.1 to 1 as studied in the main paper.
Scatter plot of Unlearning vs Utility deletion fraction = 0.5: To compare the unlearning- utility tradeoff, we fix 0.5 obtain an intermediate slice. We plot Acceorr vs Accyotain for the model produced by each hyperparameter run.
B.3.1 Unlearning across deletion fraction
Scrub: Figure 7 shows Scrub leads to gradual improvement in unlearning as increases, across hyper- parameter values. Figure 8 shows that only at high values of a, for specific hyperparameter values (high sensitivity), Scrub leads to good unlearning.
Figure shows that SSD never able outperform the original model's Acccorr in the IC evaluation. Figure 10 shows that for specific hyperparameters, SSD performs good unlearning at all values of a. The unlearning ability of SSD highly sensitive to hyperparameters.
SSD:
20

---

Published in Transactions on Machine Learning Research (10/2024)
0.8
0.4
0.6
0.3
Acfour
Acteov
0.2
0.4
0.1
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.6 Deletion fraction a
0.4
0.8
1.0
Deletion fraction
Figure 9: Unlearning performance of SSD across hyperparameters on the IC evalua- tion at different a.
Figure 10:
Unlearning performance of SSD across hyperparameters on the poisoning evaluation at different a.
0.58
0.24
0.56
Abkcom
0.54
AcC.ur
0.22
0.52
0.20
0.50
0.48
0.18
0.915
0.920
0.925
0.930
0.907
0.908
0.909
0.910
0.911
0.912
AcCystain
AcCyctai
Figure 11: Unlcarning-utility tradcoff of Scrub across hyperparameters the IC evaluation.
Figure 12:
Unlcarning-utility tradcoff of
Scrub across hyperparametens the poi- soning evaluation.
B.3.2 Unlearning-Uti Utility Tradeoff
Figures 11, 12 show that Scrub actually not too sensitive at hyperparametors, and the right values of hyperparameters (for the ones tuned) do not sacrifice unlearning for utility. On the other hand, Figures 13, 14 show that SSD is quite sensitive to hyperparameters. While there is positive correlation between unlearning and utility, surprisingly, at the highest levels of utility only few hyperparameter sets lead to high unlearning.
21

---

Published in Transactions on Machine Learning Research (10/2024)
0.8
0.4
0.6
0.3
AcCorr
Ancuer
0.2
0.4
0.1
0.2
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
AcCrdtin
AcGedain
Figure 13: Unlearning-ntility tradeoff of SSD across hyperparameters on the IC evaluation.
Unlearning-ntillty tradeoff of SSD across hyperparameters on the poi- soning evaluation.
Figure 14:
22


Reviews:
[{"review_id": "f5CAs444EU", "reviewer": "Reviewer_TDvn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n- Motivating the challenge of correcting the trained model when limited information is available about the manipulation, including its nature, extent, and the specific corrupted data.\n- Being well-written and easy-to-follow\n\n### Weaknesses\n- Limited contribution   \n- Limited experimental evaluation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript formalizes a new concept related to a potential application of machine unlearning: addressing data integrity issues. The concept introduced, called \"corrective machine unlearning,\" can be used to mitigate the influence of manipulated data in a pre-trained model, even when only a small portion of the corrupted data is available, and the type and extent of the corruption are unknown. It discusses the differences between corrective machine unlearning and privacy-oriented machine unlearning. Additionally, it experimentally demonstrates that some existing unlearning algorithms perform poorly in scenarios where both labels and features are manipulated or when only labels are corrupted.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Although the authors effectively highlight the problem, they do not propose an initial solution or offer any potential suggestions to address it. Proposing an initial approach to tackle the described problem would significantly enhance the value of this work.\n\n2) On the one hand, the formalized concept of \"corrective machine unlearning\" is compared with privacy-oriented machine unlearning, highlighting key differences in their goals, applications, and gold standards. On the other hand, some state-of-the-art privacy-oriented machine unlearning algorithms are applied to tackle the corrective machine unlearning problem. The results, unsurprisingly, indicate that these algorithms are ineffective at correcting corrupted data. Given the assumptions made when introducing the concept of \"corrective machine unlearning,\" how would the authors motivate the performed experiments? Prior to designing the experiments,  did they anticipate that existing machine unlearning approaches would be capable of addressing the corrective machine unlearning problem, or were the experiments primarily intended to confirm the differences they highlighted between machine unlearning and corrective machine unlearning?\n\n3) In the experimental evaluation, there is no gold standard to target. The assumed setting is realistic, and a gold standard does not exist. Specifically, there is no approach for correcting the side effects of data manipulation without first identifying all the corrupted data or understanding the type and extent of the manipulation.\n\n4) Moreover, the evaluation metrics require further clarification. The two metrics used for evaluation are corrective accuracy and the average change in retain accuracy over 10 different fractions of identified manipulations (alpha=[0, 0.1, 0.2, .. 1]). However, it is unclear how the latter metric should be interpreted when comparing the performance of the studied approaches and which comparison method is more effective according to this metric. Additionally, the corrective accuracy is the weighted average of the accuracy on a validation set and the accuracy on the identified set of manipulated data. Assuming all the corrupted data is identified (alpha=1), the gold standard would be the exact unlearning algorithm which implies the corrective accuracy will be the average of the validation accuracy and the forget accuracy. However, based on the assumptions made in the “corrective” unlearning problem, the fraction of identified manipulated data is less than 1. The use of this evaluation metric in the absence of a gold standard requires justification, as unknown factors—such as the type of manipulation—can impact this metric.\n\n5) On page 4, section 2.3, in the paragraph titled “Different Goals,” it is stated that “The goal of privacy-oriented unlearning is to remove untampered but sensitive user data...”. This statement can be misleading and requires clarification as the goal of privacy-oriented is removing the influence of sensitive user data, not the data itself.", "broader_impact_concerns": "I have no concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "svQdIWWCKW", "reviewer": "Reviewer_tFZW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper considers a practically relevant problem: fixing a model trained with manipulated data, where some, but not all, of the manipulated data is known. This scenario is of growing interest, especially as machine learning models are increasingly being trained on massive amounts of web data.\n- The paper shows that most of the standard *Machine Unlearning* techniques are ineffective.\n\n**Weaknesses:** \n\n\n- **Major:** The paper looks at the problem from the perspective of *Machine Unlearning* and hence only evaluates *Machine Unlearning* baselines. There are two issues with this:\n    1. It is unclear why one should take a *Machine Unlearning* perspective to the problem. One could take other possible perspectives to mitigate the effects of manipulated data.\n    2. Are *Machine Unlearning* baselines the only solution to mitigate the effects of manipulated data? What about other solutions that do not fall under the *Machine Unlearning* umbrella? For example, other methods to mitigate poisoning attacks such as those in the \"*Learning from manipulated Data*\" part of Section 4.\n\n- **Minor (easily fixable):** In section 2.2, the data domain is $\\mathcal{X} \\times \\mathcal{Y}$. But $S_{tr} \\subset \\mathcal{X}$ only refers to the data and ignores the labels. Unless the reviewer is missing something, this does not look correct. Shouldn't it include the labels, too, since they are necessary for retraining/fine-tuning? Right now, it reads as if one does not need labels for Corrective Machine Unlearning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of post-training mitigation of the negative impacts of identified manipulated training data. To address this problem, the paper takes the perspective of *Machine Unlearning* and refers to the proposed variation as *Corrective Machine Unlearning*. The paper evaluates the effectiveness of different *Machine Unlearning* techniques for this purpose and finds that most are ineffective.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It is unclear whether the paper is about mitigating the harmful effects of manipulated training data or *Machine Unlearning*. The reviewer acknowledges some similarities to *machine unlearning*. Still, the problem in the paper stands independently (based on differences stated in Section 2.3), and taking a *machine unlearning* perspective feels forceful and unnecessary. Could the authors provide some clarification?\n- In the reviewer's opinion the goal of \"machine unlearning\" is already corrective in nature. It is unclear why \"Corrective Machine Unlearning\" is the appropriate description for the problem considered in the paper.\n- This reviewer is unclear about the paper's precise claims and why they are important. As such, it is challenging to evaluate the claims. Perhaps the claims are implicitly distributed in the paper, but explicitly listing them would improve the paper and allow the reader to assess them.\n- The last line of Section 4 suggests that Yoon et al. (2023) is the closest related work. Can you clarify the similarities and differences to that paper and evaluate their approach?\n- Make the notation in the problem definition more precise (see minor weakness above).", "broader_impact_concerns": "The paper seeks to mitigate the harmful effects of data poisoning. As such, there are no direct broader impact concerns. There could be secondary impacts where the proposed solution could potentially amplify biases, which the authors acknowledge.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7GSGv0iuhY", "reviewer": "Reviewer_ZM6o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n\nS1. It is very important to investigate different unlearning applications other than privacy. Particularly, data integrity is becoming more important for training ML models and the assumption that developers might only identify a subset of the manipulated data seems reasonable. \n\nS2. The paper is well-written and structured. It is very easy to follow and understand and the experimental details are clear. \n\n\n**Weaknesses**:\n\nW1. I find the argument around the 'gold unlearning' method inaccurate and misleading. The retrain-from-scratch oracle has a clear definition which is to train a new model only using the remaining data. In your setting, when $\\alpha<1$, this baseline is trained on the remaining data plus a part of the forget-set that is not identified. Therefore, it does not have the same definition. So, I don't think it is appropriate to call this 'Exact Unlearning' in your experiments. I think it is important to make this distinction to avoid confusion. \n\nW2. The introduced problem setup in this paper is very difficult. As the results show, only SSD can achieve unlearning in limited cases with small $\\alpha$, and even that is with a big drop in utility. This makes me question the claim that this is sufficient evidence to show this problem setup is 'tractable'. To show tractability, more theoretical and empirical evidence is required. To be more concrete, if there exists two poisoned data examples during training that are completely unrelated to each other (for e.g. belong to very different classes), would it ever be possible to unlearn both by only accessing one of them? \n\nW3. The problem formulation in Sec. 2.2, is not accurate. The space of features, $\\mathcal{X}$ is defined separatedly from the space of labels, $\\mathcal{Y}$. However, $S_m \\subseteq S_{tr} \\subset \\mathcal{X}$ is said to be \"training samples manipulated by the adversary, either by modifying features, associated labels, or both.\" \n\nW4. When discussing model utility, is Eq. (2) expectation over both training examples and test examples? $(x,y) \\sim P$ implies that. If that's the case, it would be better to separate the retain accuracy from the test accuracy as they show different information and averaging them hides important facts. \n\nW5. The selected baselines are mainly designed for cases where the entire forget-set is identifiable and accessible. For your problem setup, it would be better to use other baselines that have a more relaxed assumption that make them a better match. Examples are [1, 2, 3]. Also, the model architectures (only resnet family) and the datasets (cifar and PCam) are very limited, making it difficult to make general conclusions. \n\nW6. Regarding the figures: Figure 1 doesn't seem to be very informative or clarifying. The basic setup is clear from the text. it can be omitted. In Figure 2, it is better to present a table separately with a Table caption. Also, it would be better to use the absolute value of the differences and clarify that the magnitude of the difference is the subject and one should look for a lower difference. \n\nW7. It would be a good idea to perform some sensitivity/ablation study of the baselines used, at least in the appendix. For example, SCRUB uses knowledge distillation with an annealing temperature softmax. It would be interesting to see whether increasing the temperature for the max-steps (i.e for the forget-set) would change the results.\n\n\n[1] Chundawat, V.S., Tarun, A.K., Mandal, M. and Kankanhalli, M., 2023. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 18, pp.2345-2354.\n\n[2] Chen, R., Yang, J., Xiong, H., Bai, J., Hu, T., Hao, J., Feng, Y., Zhou, J.T., Wu, J. and Liu, Z., 2024. Fast model debias with machine unlearning. Advances in Neural Information Processing Systems, 36.\n\n[3] Chen, Z., Wang, J., Zhuang, J., Reddy, A.G., Silvestri, F., Huang, J., Nag, K., Kuang, K., Ning, X. and Tolomei, G., 2024. Debiasing Machine Unlearning with Counterfactual Examples. arXiv preprint arXiv:2404.15760.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a specific use-case for machine unlearning, termed corrective machine unlearning, where the goal is to remove the effect of 'manipulated' training data from a trained model by only having access to a part of the manipulated data, and without any knowledge about the manipulation mechanism. The paper uses two methods to pollute the training data, namely, via a classic poisoning attack, and via a class confusion method. The paper shows that SOTA unlearning algorithm including the gold 'retraining from scratch', is ineffective for such a setting.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes: According to what I discussed above, I think the paper needs to improve/fix the follwoing:\n\t\n \n -- Add the following to the experiments:\n\n Related baselines\n\n more datasets\n\n more model architectures \n\n Sensitivity analysis of the baselines where applicable\n \t\n-- report utility using retain accuracy and test accuracy separately.\n\n-- Omit Figure 1 and and split Figure 2.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In my view, this paper meets the bar for acceptance to TMLR as it studies an interesting and important topic, and makes a valuable distinction of a relevant and interesting "subproblem" of unlearning from other applications / subproblems, accompanied by a thorough investigation of the suitability and performance of existing unlearning methods for this newly-identified subproblem of "corrective unlearning". As far as I can tell, the claims that the authors make are substantiated by solid evidence, with the potential exception of one relating to tractability, as a reviewer brought up, that I will suggest a minor revision for.

Reviewers found that the paper "considers a practically relevant problem" (Reviewer tFZW), is "well-written and easy-to-follow" (Reviewer TDvn), "well-written and structured" (Reviewer ZM6o) and found that "the experimental details are clear" (Reviewer ZM6o). During the rebuttal, the authors have addressed several reviewer concerns through discussion as well as additional experiments.

While some reviewers found that the experimental results "don't show a significant or novel finding", or felt that some of the conclusions were not surprising, I respectfully disagree with this opinion. I personally (being an expert in unlearning myself) would not have been able to foresee which out of the existing methods would perform best on this type of problem, at what utility cost and under which circumstances. I view it as being very valuable to report on these findings through an empirical investigation. This is important groundwork for future work building on these benchmarks, for instance, and the authors showed during the rebuttal that follow-up work already improves upon prior unlearning methods on their proposed setup. I agree with reviewers that the paper could have been strengthened if a new solution was proposed that worked well for corrective unlearning, but based on TMLR's criteria, the paper in its current form already meets the bar for publication and represents a valuable stand-alone contribution already in my opinion. I partially agree with concerns raised about tractability claims, see below.

Requested minor revisions:

- I agree to some extent with reviewer concerns raised about the claim regarding the "tractability" of the problem. Depending on the relationship ("representativeness") of the identified forget set to the overall manipulated data, I can see variations of this problem that are very challenging, and for which we don't have evidence of tractability as far as I understand (theoretically nor empirically). Can the authors please avoid using the word "tractability", but rather make a more concrete claim, e.g. something along the lines that, for the variations of the problem that are studied in this paper, there is encouraging evidence that progress can be made, even if partially, and discussing concretely which metrics see progress (citing, e.g. the partial success of the SSD approach) rather than making more general claims about tractability. It would be great to also add an explicit note in the limitations section that not all possible instantiations of corrective unlearning may be tractable. This doesn't diminish the relevance of the work (and I agree with the authors that this is a valuable area for future research), but it is important that the claims are as accurate and the relevant nuances are discussed appropriately.

- The authors state that "Our findings indicate that popular unlearning methods, even the traditional gold standard of retraining on the remaining data (RewoD ), fails to enhance accuracy on manipulated domain samples unless nearly all of the manipulated data is identified". I view this phrasing (here and in some other parts of the paper) a little misleading: the gold standard of retraining could in fact also be an ideal solution to this problem (but only if the full set of manipulated samples are identified). Similar to the argument of Reviewer ZM6o for using the term "exact unlearning", it doesn't feel right to refer to retraining on a "retain set" that still contains some manipulated data as a "gold standard". I would rewrite this sentence (and other similar occurrences) to emphasize that i) retrain-from-scratch is the gold standard for the version of the problem where alpha = 1, and ii) for alpha < 1, it should no longer be considered the "gold standard" and may perform worse than other unlearning methods.
Decision: Accept (poster)