Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted data
Authors: Stefan Schoepf
Michael Curtis Mozer
Nicole Elyse Mitchell
Alexandra Brintrup
Georgios Kaissis
Peter Kairouz
Eleni Triantafillou
Venue: ICLR 2026 Poster (2026)
Keywords: machine unlearning
corrupted data
data poisoning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We present a taxonomy for corrupted data unlearning tasks along two dimensions: regularity and discovery rate, and show that no prior method succeeds in all but a slice of that space. Our method is the first to perform strongly across this space.

Abstract: Machine unlearning is studied for a multitude of tasks, but specialization of unlearning methods to particular tasks has made their systematic comparison challenging. To address this issue, we propose a conceptual space to characterize diverse corrupted data unlearning tasks in vision classifiers. This space is described by two dimensions, the discovery rate (the fraction of the corrupted data that are known at unlearning time) and the statistical regularity of the corrupted data (from random exemplars to shared concepts). Methods proposed previously have been targeted at portions of this space and, as we show, fail predictably outside these regions. We propose Redirection for Erasing Memory (REM), whose key feature is that corrupted data are redirected to dedicated neurons introduced at unlearning time and then discarded or deactivated to suppress the influence of corrupted data. REM performs strongly across the space of tasks, in contrast to prior SOTA methods that fail outside the regions for which they were designed.

Published as a conference paper at ICLR 2026
REDIRECTION FOR ERASING MEMORY (REM): TOWARDS A UNIVERSAL UNLEARNING METHOD FOR CORRUPTED DATA
Stefan Schoepf University of Cambridge
Michael C. Mozer
Nicole Mitchell Google Research
Alexandra Brintrup University of Cambridge
Google DeepMind
George Kaissis Google DeepMind
Peter Kairouz
Eleni Triantafillou
Google Research
Google DeepMind
ABSTRACT
Machine unlearning is studied for a multitude of tasks, but specialization of un- learning methods to particular tasks has made their systematic comparison chal- lenging. To address this issue, we propose a conceptual space to characterize di- verse corrupted data unlearning tasks in vision classifiers. This space is described by two dimensions, the discovery rate (the fraction of the corrupted data that are known at unlearning time) and the statistical regularity of the corrupted data (from random exemplars to shared concepts). Methods proposed previously have been targeted at portions of this space and, as we show, fail predictably outside these regions. We propose Redirection for Erasing Memory (REM), whose key feature is that corrupted data are redirected to dedicated neurons introduced at unlearn- ing time and then discarded or deactivated to suppress the influence of corrupted data. REM performs strongly across the space of tasks, in contrast to prior SOTA methods that fail outside the regions for which they were designed.
INTRODUCTION
Unlearning is the problem of removing the effect of subset of training data from a trained model (gguyen et al.] (0022. In this work, we consider a scenario where after having already trained model on a dataset, we discover that a subset of the training data was accidentally mislabelled, low quality, or manipulated by an attacker, causing the model to make mistakes or exhibit unwanted behaviours. The goal of unlearning (UL) to post-process the trained model to efficiently remove (the effect of) that corrupted data in order to restore the correct predictions and behaviours.
While the problem of unlearning has attracted significant attention (Triantafillou et al. 2024 Hayes et al. 2024 Goel et al. 2024), and plethora of methods have been proposed, there is still a lack of scientific understanding of the behaviours of these methods on different types of UL tasks, with only early work in this direction (Zhao et al. 2024 Goel et al.] 2024). This lack of understanding when established unlearning methods fail or succeed is a fundamental blind spot as it hinders research progress and can lead unpredictable failure in practice, shown in this work.
To address this, our first contribution is a taxonomy of tasks for unlearning corrupted data shown in Fig. [ Our taxonomy is based on the identification of two dimensions along which the behaviours of unlearning algorithms differ substantially. The first is the discovery rate, the fraction of corrupted data that have been identified and can be utilized by the unlearning algorithm of corruptions. Gool et al. (2024) previously studied the effect of discovery rates on unlearning performance, finding that performance of algorithms developed for the full discovery case drops off suddenly as the discovery rate is lowered continuously. We take this study a step further by identifying second dimension that affects performance significantly, both on its own and through interaction with the discovery rate. The second dimension is the statistical regularity of the cor-
remove the effect
"Wor done during the author's internship at Google DeepMind. PhD funded by EPSRC DTP (FP/W524633/1)) Corresponding author; sx2823 cam.ac.am.kkk

---

in Published as a conference paper at ICLR 2026
Corrupted Data Regularity
(b) Retrained from scratch
(e) SCRUB
(c) Potion
(f) Gradient Ascent
A
Corrupted Data
0%
100%
Discovery Rate
(a) Framework for corrupted data unlearning tasks with prior work areas highlighted
(d) ETD
(g) REM (ours)
Figure I: We present new taxonomy of unlearning tasks in of two dimensions: the regularity and the discovery rate of the corrupted data we wish to unlearn. The highlighted areas in (a) show tasks studied in prior work. Subplots (b-e) show an aggregate metric of unlearning performance (see Section 2 of methods for different discovery rates >0% (x-axis) and regularities (y-axis), in- stantiated via the benchmarks of (Goel et al., 2024). Darker color is better. Prior methods succeed only in slices of this 2D space, mainly failing along the regularity axis. Representative prior work: [A] Maini et (2023). [B] (Schoepf et al. (2024b), Chundawaa et al. (2023); Kurmanji et al.
(2023), [D] Goel et al. (2024). [E] Zhao et al. (2024) Foster et al. (2024): Chundawat et al. (2023)
Kurmanji et al. (2023).
Chundawat et al. 2023); Kurmanji et al. (2023
rupted data, tracing spectrum from "spurious" corruptions (such random mislabeling), towards structured corruptions that systematically affect related data points (such as a poison trigger that redirects all images which it appears to pre-specified class label). Our key finding is that, in our 2D space of tasks, different state-of-the-ard (SOTA) methods succeed different slices, but each fails predictably and catastrophically everywhere else. These methods therefore risky practice when the task specification is not known.
To address the need for universal method for unlearning corrupted data that covers the 2D space, we present our second contribution: Redirrccton for Erasing Memory (REM), new unlearning al- gorithm that is the first to perform strongly across our task space. REM employs a novel mechanism, inspired by the memorization mitigation method Example-Tics Tied Dropout (ETD) (Maini et al. 2023), redirect the corruption model by REM) that can then be conveniently dropped off or deactivated to unlearn even undiscov- ered corrupted data of any regularity. We benchmark REM against SOTA unlearning methods as well as ETD on low, medium, and high regularity unlearning tasks for discovery rates ranging from 10% 100%. Experimental results on CIFARIO (Krizhevsky et al. 2009) and Street View House Numbers (SVHN) (Netzer al., (011) with ResNet-9 (He et al.] (2166) and Vision Transformer (ViT) (Dosoviskkyy et al. (202]) as well as different optimizers show that REM is the only method perform well equally well across the entire 2D space.
dedicated part of the network (newly initialized capacity added the
2 PROBLEM DEFINITION AND BACKGROUND
Machine unlearning applications range from privacy protection of individuals (Neel al., 2021 Sckhari et al.) 2021 Hayes et al.) 2024) to correcting errors in the model due to corrupted training data (Goel et al. 2022] Kurmanji et al. 2023 Goel et al.] 2024), each which has different goals and evaluation metries. In this work, focus unlearning corrupted data for classification tasks.
Practical examples of corrupted data can range from simple data entry errors (Tanno et al. 2022 Schoepf et al. 2024a to large scale data poisoning attacks (Carlini al. 2024 Zhang et al.) 2024b).
Problem definition. Let Drr denote a training dataset, and Diea a held-out dataset of the same distribution. Further, let A denote training algorithm, and = A(D) the weights of neural
2

---

Published as a conference paper at ICLR 2026
network that are learned by applying A on D. We consider scenarios where some training data De C Dr is corrupted. Let Dj S De denote the forget set comprised of corrupted data discovered after training. We refer to the rest of the training data, Drr \ Dj D, the retain set. This formulation reduces to the most commonly-studied variation of unlearning when Dj- = D2 i.e. the full set of data that we wish to unlearn is provided in the forget set. In that case, straightforward but computationally expensive solution is to retrain from scratch to obtain a new model 0,- A(D,) that not affected by corrupted data. For "partial discovery", defined as Dj C De. O, not solution since undiscovered corrupted data De Di 'D Pundisccovered is part of Dr so re-training on it does not eliminate the corruptions (Goel et al.] 2024). The problem we are interested in is to design a (A, UI) pair of a learning algorithm A and post-processing (unlearning) algorithm 2 such that, when forget set Dr identified, 21 can efficiently post-process the trained model O. = A(Drr) to yield an unlearned model 0g = (0.0. Dr) that does not suffer from failure modes caused by De. while having high utillity. Note that some choices of A may facilitate the success of 2 to post-process for unlearning. We assume for simplicity that Ul can access Dr, is common in the literature.
Memorization and statistical regularities. Feldman (2020) refers to an example æ as being "mem- orized" by learning algorithm A in a training set Du. if, in expectation, models obtained through recipe ADu,) are much more likely to make correct predictions on 2' compared to models obtained through recipe A(Drr) \.) Intuitively, x is memorized if it is required in D,, to be correctly predicted. This will be the case for atypical/ irregular examples (Feldman & Zhang 2020), whereas typical / regular examples that are similar to many others in the dataset may be predicted correctly simply due to generalization (i.c. being learned through other samples), without having to be part of the training set. Similarly, [Ding et al. 2020) define a consistency score of example x to measure its statistical regularity via the expected accuracy on that example under models obtained by training on different subsets of Drr that exclude x. Building upon this notion, we identify the statistical regularity of the corrupted data as a key factor influencing behaviours of unlearning algorithms.
Example-Tiee Dropout (ETD) is a training recipe proposed by Maini et al. (2023) that separates model parameters into those that intend to capture "generalizable information" that is shared be- tween many data points, and those that intend to "memorize" " information that is specific to one or a few data points. This separation is made within every layer of neural network. During training, the computational paths of all data points pass through all of the generalization neurons (that is, the generalization neurons are never dropped out), but when it comes to the memorization partition, each example only passes through its dedicated path, based on an example-tien dropout mask that is randomly determined ahead of training and then fixed. This allows different pathways in the mem- orization partition to be "owned" by different examples, enabling those examples to encode their potential irregularities
their dedicated paths. On the other hand, example-specific irregularities won't be as easily encoded in the generalization neurons, since those are shared by all examples. At inference time, the memorization neurons are dropped out, reducing or eliminating example-spectis information. A similar approach has been applied by Ghosal et al. (2025) to prevent verbatim mem- orization in LLMs. We propose ETD as an UL algorithm, which can be seen as setting A as ETD training followed by dropping out all memorization neurons post training, and setting /t a no-op.
3 RELATED WORK
Unlearning methods. We build upon the benchmarking selection of Goel al. (2024) to cover dif- ferent state-of-the art approaches to unlearning and extend it with recent advances and overlooked baselines. Goel et al. (2024) included Bad Teacher (BadT) (Chundawat et al. 2023). SCRUB (Kur-) manji et al. 2023). ). Selective Synaptic (SSD) (Foster et al. 2024), fine-tuning, and retraining from scratch. BadT randomizes labels on Dj by distilling from randomly initialized network to induce forgetting and distilling the remainder of the samples D, \ Dj. SCRUB alternates between step of distillation away from the original model on D, and step towards the original model on D. for model utility preservation on the remaining data. SSD determines disproportionately important parameters for Dr compared to Drr in the model via the Fisher Information Matrix and dampens these parameters to induce unlearning. We replace SSD in our benchmarks with Potion (Schoepf et al. 2024b) which builds upon SSD and is the SOTA in poison unlearning as defined by Goel et al. 2024) (see Fig. D. Potion iteratively increases the number of parameters in the model that it modifies analogously to SSD until unlcarning of the poison trigger occurs. Potion stops unlearning as soon as it detects that the accuracy of the model on the forget set drops below preset threshold,
3

---

Published as a conference paper at ICLR 2026
indicating that the poison has been unlearned. Fine-tuning continues to train the model on only Dr. relying on catastrophic forgetting to remove Dr. Retraining from scratch trains a new model using only Dir \ Dj. Ascent performs gradient ascent on Dr. NPO (Zhang et al. (024a) is an alignment- inspired loss function for unlearning that overcomes the catastrophic collapses of Gradient Ascent. This is achieved via reference- model based loss calculation.
Investigation of unlearning limitations. Goel et al.
(2024) studies the effect of partial discovery For unlearning poisoned data, they demon- 2024g fail to remove the effect of the poison when
on unlearning method effectiveness shown in Fig.
strate that all methods apart from Foster et al. only a subset of the poisoned examples is given (partial discovery). They also consider another setup, inter-class confusion, where samples from two classes are swapped with each other, where no method shows satisfactory performance in the partial discovery case. iZhao et al. (2024) iden- tify interpretable characteristics of forget sets that substantially affect the difficulty of the problem, but they don't i consider the statistical regularity of the forget set in their investigation. |Rezaei et al.) (025) show that standard unlearning methods designed for erasing knowledge may fail at restoring knowledge present prior to corruption. This relates to our metrics of healing, where we are interested in redirection to the correct label, rather than just avoiding predicting the incorrect (e.g. poisoned) label. However, their investigation is in the context of LLMs and they don't t establish a distinction between tasks of different discovery rates or regularities.
4 TAXONOMY OF UNLEARNING TASKS AND METHOD LIMITATIONS
Taxonomy. Our first contribution is to propose new laxonomy of unlearning tasks according to two dimensions: the discovery rate and the statistical regularity of the corrupted data that we wish to unlearn. The discovery-rate dimension was previously discussed and empirically investigated in Goel et al. (2024) but is insufficient to explain model failures, as methods such as Potion (Schoepi) ctt al. 2024b) excel at some low discovery tasks but then fail at other full-discovery tasks. We further identify regularity
key dimension that affects the behaviour of unlearning algorithms both on its own and in conjunction with the discovery rate. Our key observation is that, as shown in Fig. [ each prior method may excel on a slice of our 2D space of unlearning tasks but fails catastrophically everywhere else. We discuss the limitations of different algorithms below.
Discovery rate. Unlearning methods that are designed for the traditional unlearning setting of full discovery fail catastrophically for lower discovery percentages as described by Goel et al. (2024). Retraining from scratch and unlearning methods that utilize the retain set, e.g. SCRUB, fail due to requiring a clean retain set for utility preservation, but in the partial discovery case, the retain set contaminated and thus leads to reintroduction of corruptions (see Fig. (b, e)). Methods such as Potion and SSD that only use Dj cause significant model utility damage due to the missing repair step on D. (Goel et al. 2024; Foster et al., 2024).
Regularity refers to the self-similarity of the corrupted data De. Informally, we say that corruption has low regularity if the corrupted data does not share any structure or common patterns. For exam- ple, a random mislabelling of randomly-chosen subset of examples is low-regularity corruption because the self-similarity between samples that end up with the same label change is low. On the other hand, poisoning attack that introduces trigger to each image whose label it wants to redi- rect results in high-regularity corruption since all corrupted images share the visual pattern of the poison trigger and are all labeled in the same way. In the context of the commonly studied problem of full discovery unlearning, class unlearning (Golatkar et al. 2020) and random subset unlearning (Chundawat et al. 2023) are high and low regularity tasks, respectively. Several measures could be used to formalize regularity. We advocate for the consistency score (C-score) of Jojan et al. (2020). that measures the expected accuracy of a model on held-out samples from the training set. Their precomputed CIFAR10 scores align with the regularity rankings of our tasks (see Fig. 13 in Jiang al. (2020)). Adapted De, C-scores can be computed with n 0, DD] for instances xr and labels y. where E" is empirical averaging over samples (Jiang et al. 2020):
Co, (I. v) = E, E, D&D, [P(((z v)1) = y)]]-
Failure along the regularity dimension for ETD is shown in Fig. (d). where ETD succeeds at lower regularity tasks but fails at high regularity tasks. This is because information from regular sam- ples, by design, reside in the generalization neurons of ETD, while low-regularity information (e.g.
4

---

Published as a conference paper at ICLR 2026
example-specific peculiarities) is encoded in the memorization neurons that are dropped to induce forgetting. On the flip side, the Potion method excels at unlearning high regularity poisoned data, which is what it was designed for. However, it fails catastrophically at lower regularity unlearning tasks. This is because Potion is built on the assumption that the data that we wish to unlearn resides in concentrated (small) set of locations in the network; a hypothesis that is likely hold for high regularity data (e.g. some specific neurons may be disproportionatlly responsible for encoding the poison trigger) but unlikely to hold for low regularity, where the corrupted data share no charac- teristics. making it unlikely for them to be stored in similar locations. Gradient Ascent (Fig. | (f) also performs poorly at lower regularity tasks compared to high regularity tasks. We hypothesize that this is because the information of low regularity tasks is spread across wider set of parameters, making it harder to erase without overly damaging the model utility using Gradient Ascent.
Interplay between regularity and discovery. As discussed before, traditional unlearning methods for full discovery are often based on fine-tuning on the retain set. For partial discovery, this is problematic as the retain set contains undiscovered corrupted data that are (re)introduced during fine-tuning (or retraining). However, interestingly. the degree of this (re)introduction from the partial set of corrupted data that lives in the retain set is function of the regularity, with highly regular corruptions leading to amplification of this reintroduction. This is because for regular data, the presence of a few instances in the retain set suffice (re)introduce the general shared pattern. For instance, the association that the poison trigger should redirect to a specific output is one that can be learned from few poisoned examples, allowing the model to then predict the same output for other poisoned data that have the same trigger, without those being in the retain set, due to generalization. On the other hand, contamination of the retain set by low regularity corrupted data will lead to learning (or reinforcing) incorrect labels on that data but the damage won't 't spread beyond those specific examples. This is why, for partial discovery, retraining on D, (Fig. [Tb) suffers more pronounced and sudden drops the higher the regularity.
Experimental setup. We evaluate tasks of low, medium, and high regularity with ten discovery rates for each task (10%-100%). Low regularity is represented by random label swaps of n samples to an incorrect label (Maini et al., 2023). In this setting, there is no regularity linking the samples or the new labels together. Medium regularity uses the inter-class confusion setup of Goel et al. (2022) where n samples of two classes are swapped with each other. The highest regularity task uses a poison trigger to redirect to class O as defined by Goel et al. (024). We measure the accuracy of 9, on two types of data: (i) corrupted data (using the clean labels to compute accuracy), and (ii) non-corrupted data. The former is used to measure "healing". i.e. whether the label prediction has been "redirectedd" to the correct label, while the latter is used to measure "utility",, i.e. ability to predict correctly on non-corrupted data. We perform experiments on an A100 (40GB) GPU with ResNet-9 (He et al. (2166) and Vision Transformer (ViT) (Dosovitskiy et al. (022)) models, stochas- tic gradient descent (SGD) and Adam (Kingma & Ba, 2014) optimizers, and CIFARIO (Krizhevsky et al.] (2099) and Street View House Numbers (SVHN) (Netzer et al. 2011) to show generalizability across models, optimizers and datasets. GitHub:
REDIRECTION FOR ERASING MEMORY (REM)
We now introduce our core contribution, Redirection for Erasing Memory (REM), the first unlearning method that is performant across our 2D space spanning different regularities and discovery rates, without requiring specialized hyperparameter tuning for each region. The design of REM is based on the lessons discussed in the previous section: (i) In the setting of partial discovery. finetuning on D. causes reinforcing or reintroducing corruption (especially for high-regularity corruptions), but not using D, at all would lead to low model utility; (ii) ETD can eliminate the effect of low-regularity corruptions by simply dropping out the memorization partition, but this operation will not remove the effect of high-regularity corruptions, as those are encoded in the generalization partition.
Overview. REM's key innovations are follows. First, instead of avoiding the use of D, (which would lead to low utility), REM accepts that undiscovered corrupted samples in D, will unavoidably re- re-introduce comuptions but it employs novel mechanism to redirect them to dedicated part of the model at unlearning time that can then be dropped or deactivated. Second, the above is achieved through the design of novel unlearning objective using loss and masking strategy.
5

---

Published as a conference paper at ICLR 2026
Expand model 0,
Remove or using only Dj
Redirect & Repair with D
Drop redirection
Parameters relevant correct predictions
Parameters relevant for comupted precictions
Figure 2: REM performs the following steps: (i) Expand the network with randomly-initiaizzed parameters 0 og (ii) Remove the corruptions out of 0, with a SOTA unlearning algorithm on for that does not use Dr, avoiding reintroduction in do., but at the expense of utility; (iii) Repair utility by fine-tuning 9o with Pu. using novel Redirection strategy to steer any reintroduction of corruptions caused by the inclusion of DD to the add-on parameters fog; (iv) Drop out Pog.
Let 0o, = A(rr.) denote the parameters of the original model, trained using A on Du. Unlike ETD, REM's choice of A can be any standard training procedure. REM then applies post-processing U on top of 0, to unlearn corrupted data by redirecting the effect of corruptions to newly-added parameters Pgg that are then dropped. We emphasize that, while this design draws inspiration from ETD, is a fundamentally distinct methodology: while ETD attempts at training time to partition the network into generalization and memorization, REM attempts at post-processee network into parameters that are unaffected by the corruptions and ones that capture all corruptions. Furthermore, REM adopts a unique masking strategy to capture high regularity corruptions that cannot be captured by ETD. We detail each REM step from Fig. 2 below and in Alg[]
time to partition the
Expand. As first step, REM expands the network with additional capacity, initialized randomly. These new parameters, denoted as fog. can be seen as a placeholder at initialization, to which REM will attempt to redirect the corrupted data, as discussed in a later step. While REM allows complete freedom in the architectural design of 0o, and 0og. we adopt simple approach here that enables direct comparison with ETD: we expand the number of channels in each convolutional layer. The structure and capacity of each of 6,,, and 6o, correspond directly to that of ETD's 's generalization and memorization parts. However in REM, 6o, is obtained by standard training (so it doesn't 't capture only "generalization") while 0,, is
-nnttalliii expansion added at postprocessing time.
Remove corruptions from 0,, (step 2 of the pseudocode). We achieve this via Negative Prefer- ence Optimization (NPO) (Zhang et al. 2024a), a SOTA unlearning method in the NLP domain which we translate to our classification problem. NPO is chosen over Potion and Gradient Ascent, the other UL methods available that do not utilize D,, because Potion destroys model utility to an unrecoverable level at lower regularity tasks (see Fig. (c)) and NPO has demonstrated better per- formance restoring or healing knowledge, as opposed to removing knowledge in the NLP domain (Rezaci et al., (025), which is closely related to our goal of healing corrupted data. As research progresses, better UI. methods that do not utilize D, can replace NPO to boost REM performance further. NPO, adapted to classification by using the cross-entropy loss Lc: = Σe= ye log(ūe)) is shown as Cremoven., in Eq. 2 and used in step 2 of Alg. [ Note that independence from the retain set during this step essential avoid reintroduction of corruptions in 0, The downside of this is utility degradation which we address in the next step. The stop condition in Alg. T is adapted from Schoepf et al. (2024b), which shows that unlearning does not take effect gradually but suddenly as forgetting occurs. The parameter 7, as shown in [Schoepf et al. (2024b), is not highly sensitive and should be chosen higher than random chance but lower than model utility.
Repair utility and redirect corruptions into Ooz instead of relearning in O.,- At this point, we have removed the effect of conuptions from do at the expense of utility, via having unlearned on 0o, using an algorithm that doesn't 't use D, We now aim to restore utility using Dr. This operation is risky since, for partial discovery, Drr contains undiscovered corrupted data. We therefore design a loss function for this step that can benefit from the clean data of Drr while attempting to redirect the corrupted data of De into f.,, which will then be discarded.
To that end, design masking strategy for 6o, analogously to ETD's masking strategy in its memorization partition. In particular, every example in Dr will receive its own randomly deter- mined mask in l lzg. All examples in D, are assigned the same (randomly-deeermined) mask each

---

Published as a conference paper at ICLR 2026
Algorithm Redirection for Erasing Memory (REM)
Require:
Training dataset Dr and forget set Dr C Dir. Learning rate e, total epochs max epochs
Require: Dj removal stop condition: Acc(D;) <
Step 1: Add additional capacity 0. to the existing model 0a1 for redirection
while current epoch I max epochs do
Step 2: Remove Dj from the 0o1 part of model
while (C)(DD) > 7 do
2.1 Compute loss Cremsweu,, (Dr) using only the 0,1 part of the model (see Eq.
2.2 Update 0o1 model part with loss Laup2 Cremovesnn
end while
Step 3: 0. model utility repair and De redirection into 0,2
3.1 Compute loss Fredirect using the 0,1 U 0.2 model with Dar, where all Dr Dir samples are assigned the same mask for redirection in 0o remaining samples use random masks
3.2 Compute Cremowes,, (Dr) using only the 6,1 part of the model (see Lremovea in Eq. 2
3.3 Update model with the combined loss Lsups Leedirscte, U8o2 Lremovee., shown in Eq.
end while
Step 4: Discard additional capacity 0.e to keep only 0ot as the unlearned model.
other. Then, directly analogously to ETD, all examples pass through all of 0o2 but each example will only pass through its assigned path in 6o, based on its mask. The intent that forcing all forget set examples to share the same mask will cause the resulting path in fo, to strongly learn the corrup- tions, turning into a "channel" for corrupted data. Utilizing the same mask for all corrupted samples intuitively seems like a problem, as the model may learn generalized knowledge in the added ca- pacity when a neuron is present in multiple masks of samples that share generalizable knowledge. Specifically, when the added capacity is dropped, this generalized knowledge will be lost and model utility drops. This can indeed be true the masking strategy is active from the start of training. However, in the post-hoc unlearning setting of REM, the model is already trained. Existing general- ization will not switch over to the added capacity mask, as the path least resistance is to reuse the already present neurons in the model that contain this information. We therefore first need to remove the existing information from these neurons (the Remove step of REM) to be able to redirect them. Because we had already unlearned corrupted data from do, in the previous step (a property which we reinforce further by adding NPO to the objective of this step), we are able to direct the corrupted data of the forget set to this channel fzz-
At the same time as the redirection of corruptions, clean retain set examples can repair utility through fine-tuning 9,, which is kept clean from corruptions due to the previous removing step and the continued application of NPO on 0,, (Alg. step 3.2). The combined loss (Alg. 1 step 3.3) is shown with a as the sigmoid function:
Capp3 E log o (-81og
[CBes, UBo, (Dn))
2 log a (-Slog
[CFeas (Dt) LCEres (Dr) ))
(2)
LCEreS (Du)
B
Credireti Uouz
The reference models ("ref") correspond to the initialization of the respective weights (both 6., and fo, in the first term; only 0,, in the second). This formulation is similar Direct Preference Optimization (DPO) (Rafailov et al. 2023) with the key difference that we perform each part on different model (i.e. different set parameters active and an example-dependent forward pass).
REM on ETD. While it's not necessary, REM can be applied over an ETD-trained model. In that case, we don't 't need to expand the network further; we directly use ETD's 's memorization partition for redirecting the corruptions, and ETD's existing masks for the examples in D, (assigning a new one for examples in Dj). All other steps of REM then proceed as usual. We evaluate this variation empir- plain REM,
ically too, finding that it boosts performance on low regularity and low discovery but this comes at cost of some utility loss caused by the ETD training scheme (see appendix Fig, g for comparison of the utility of pretrained models with and without ETD at different capacities).
7

---

Published as a conference paper at ICLR 2026
Model 100% Model
Utilty Healed
60
40
20
Figure 3: Comparison of UL methods model capacity levels using ResNet-9 & CIFARIO with 1000 corrupted samples, three regularity levels and 10 discovery rates (10%-100%). REM (IDEAL) provides and upper limit with perfect knowledge of manipulated samples for mask assign- ment. ETD and REM are not reported for 100% capacity as reserve capacity is needed for f, of the model. Error bars reflect +1 SEM.
Table I: Aggregate results of methods on CIFAR 10 with ResNet-9 and ViT with SVHN (50% capac- ity each) and 1000 corrupted samples with 10 discovery levels (10%-100% and regularities. Presented in descending order of Utility x Healed (ResNet-9), Potion VIT experiment failed due to OOM (Out of memory) with A100 (40GB VRAM). Error reflects =1 SEM.
ResNet-9
VIT
ETD
UL type
Healed
Utility
Utilityx Healed
Utility x Healed
REM
81.16 + 1.62
90.54 + 0.15
73.40 + 1.43
73.27 = 0.32
REM
83.26 0.92
88.05 = 0.18
73.19 0.72
72.80 = 0.36
NPO
77.50 + 1.53
86.99 + 0.24
67.10 1.17
66.31 = 0.86
Ascent
16.59:1.44
86.40 = 0.35
65.82 + 1.08
67.70 = 0.90
SCRUB
66.95 2.82
89.45 = 0.14
59.85 2.50
55.03 = 3.24
BadT
6.824
88.13 + 0.16
58.32 1.63
52.50 + 3.13
NPO
6.98
87.59 + 0.36
56.40 2.50
65.62 + 0.88
Ascent
63.98
86.97 = 0.29
2.49
67.14 0.89
Potion
3.773
62.86 3.62
51.30
0OM
No UL
56.97
88.00 + 0.14
20.11
51,41 + 3.33
Retrained
53.61 2.73
90.46 + 0.14
48.52
56.36 3.07
SCRUB
47.09 2.58
90.71 + 0.15
42.69 2.33
55.29 3.22
BadT
46.17 0.84
90.23 + 0.18
41.60
52.40 3.08
Potion
49.39 3361
53.06 1 3.30
4.662
OOM
No UL
35.33 + 1.66
89.94 = 0.17
31.72 1.46
51.02 = 3.29
RESULTS & DISCUSSION
REM represents a leap in coverage of the 2D space compared to previous methods as seen in Fig [ setting new SOTA. Unlike prior methods, REM does not exhibit a sudden breakdown along any of the framework dimensions and therefore presents itself safe choice for use practice. All REM results use the same hyperparameters, showing generalizability across models, datasets and tasks.
Metrics. "Healed" computes the clean (i.e. uncorrupted label) accuracy on the corrupted D. inputs. Utility computes the accuracy on clean test set without any corruptions. Utility Healed provides multiplicative score, where higher better, to ensure that the method not only removes the cor- rupted data but also retains model utility, analogous to Zhao al. (2024). Figs. (b-g) are shaded using this aggregate metric. Reported errors reflect + standard error of the mean (SEM).
Method comparison. Fig, shows that REM is the only method that performs strongly all regions of our 2D space. We additionally provide aggregate numerical scores in Fig. that are computed the average of the Utility Healed metric across runs for all discovery rates and regularities. We make the following observations. For all methods, the score increase for the 50% capacity model the 100% model due to less memorization of lower regularity corruptions due
8

---

Published as a conference paper at ICLR 2026
High 13.4
89.3
89.2
89.0
89.4
89.2
89.1 89.4
89.1
89.2
89.3
High
14.2
81.3
81.3
81.2
81.6
81.3
80.8
80.8
80.5
81.6
81.6
Med.-
36.8
41.4
49.2
54.7 59.3 62
6
65.7 69.0
73
74.3
Med.-
61.8
63.9
65.8
66.
65.8 65.4
65.8 64.7
64.2
83.6
63.6
Low
62.3
63.2
06.3 71.8
72.9
73.4
77.2
Low
74.0
73.4
73.4
73.8
73.6
73.4 73.3
73.4
73.2
0.4
0.5
0.6
1.0
0.4
0.5
0.6
0.7
1.0
discovery rate
discovery rate
(a) REM on 50% capacity model
(b) REM on 50% capacity mode trained with ETD
Figure 4: Comparison of REM applied to model trained with/without ETD. The performance of the model before applying REM shown in the 0.0 discovery column. (b) shows that ETD provides an uplift in lower discovery rates for lower regularity tasks (y-axis) but comes at the cost of overall model utility (see Tab. T which harms the higher regularity and higher discovery rate performance.
to less overparameteriiatton. We observe that ETD significantly outperforms the approach of simply training a smaller original model ('no UL" in Fig. B) In fact, we find that ETD alone is strong baseline that was not previously considered. It outperforms prior unlearning methods in terms of the aggregate scores, due to the effectiveness of memorization mitigation on lower regularity tasks. Further, as also discussed by Goel et al. (2024). retraining model from scratch is not the gold standard for partial discovery tasks as the undiscovered corruption is reintroduced into the model, leading to poor performance. Potion does not perform well on this aggregate metric due to failure on the lower regularity tasks. A surprising discovery is that the simple baseline of Gradient Ascent beats many other methods baseline was believed to be weak and was even omitted from prior comparisons (Goel et al. 2024). However, note that Ascent fails across entire slices of the 2D space shown in Fig. which is hidden in the aggregation but may be problematic in practice. REM outperforms all prior metrics and even comes close to REM (IDEAL), an upper bound that has knowledge of all corrupted data when forming masks at unlearning time.
2D space in terms of aggregate score. This is surprising this
Unlearning on an ETD model. Tab. shows that REM is the best performing method when used either with or without ETD training. The choice of whether to use ETD pretraining leads to trade-off, as shown in Fig. ETD improves performance on the lower regularity tasks but its training procedure causes model utility deficit as the model used during training differs from the model inference time. Other unlearning methods show an increased combined score of Utility x Healed when combined with ETD but do not come close to REM. This is due to the shortcomings of the last used method affecting the results. For example, (i) in SCRUB, the reintroduction of the poison trigger in partial discovery settings causes failure in high regularity tasks, (ii) Potion causes significant model utility damage in the lower regularity settings negating the benefit of ETD.
versus postprocessing. As shown in Fig. 3 less overparameterrzztion (smaller capac- ity) is effective in partly mitigating memorization of corruptions during training. We report extreme cases of this in the appendix (see Fig. { showing that by restricting model capacity very low levels, no space for memorization remains, effectively solving lower regularity tasks but at the high cost of model utility. Therefore other mitigation actions during training, such as ETD for instance, should be preferred over extreme capacity restriction. Based on these insights, we argue that high regularity corruptions are the hardest to address during training. We suggest that this also applies to other domains such as NLP, where verbatim memorization can be addressed in simple ways such as with the Goldfish loss (Hans et al. 2024) but concepts (high regularity) are difficult to mitigate.
Prevention
Ablations. Tab. 2 shows that redirection of manipulated samples is essential for REM performance. Other aspects of REM are more nuanced. ETD comes with the trade-off of more healing at the cost of utility. Step 3.2 in Alg. Tincreases the healing performance at higher discovery rates as this allows for better redirection by preventing reintroduction into the 0,, part of the model. When trained with ETD this difference vanishes as it is mainly observed in lower regularity tasks (Appendix Fig. 7
Different dattse//rrhittetur//ppiiiiiee
ViT (Adam, SVHN) results in Tab. [ are in line with the ResNet-9 (SGD, CIFAR10) results with the key difference that the ViT model shows better mitigation against outliers low regularity corruptions. This increases scores as the No ULs for medium and low regularity tasks are higher than in the ResNet-9 setting.

---

Published as a conference paper at ICLR 2026
Table REM ablation using CIFARIO with ResNet-9 analogous Tab. | Redirection (3.1) using Du instead of just finetuning on D, is essential while step 3.2 only matters when the model is not trained with ETD. No of 3.1 & 3.2 simplifies NPO, as o, is unused. Error reflects =1 SEM.
Step 3.1
Step 3.2
ETD
Healed
Utility
Utility x Healed
83.70 + 0.89
88.16 + 0.16
73.69 = 0.69
81.16 + 1.62
90.54 0.15
73,40 + 1.43
83.26 = 0.92
88.05 0.18
73.19 = 0.72
78.94 + 1.67
90.55 0.15
71.38 = 1.47
J
77.31 2.07
90.39 0.14
69.82 1.85
78.83 1.45
87.17 0.21
68.68 1.26
Accuracy on corrupted data
100
Base Model
50
Base Active Mask of Expansion
25
0
1
2
4
Unlearning epochs
Figure 5: Accuracy on corrupted data for the base model (i.e. the existing trained model) and the base model plus the additional active parameters from the added mask. Epoch 0 is the starting point where no unlearning has taken place. unlearning takes place, the corruptions are redirected into the additional capacity of the model and removed from the base model. This enables the base model to be free of corruptions by dropping the added capacity after unlearning.
Redirection of corruptions. We show the effectiveness of our redirection strategy using ResNet9, 50% discovery rate, and the poison trigger task over 5 unlearning epochs. As shown in Fig. 5 the base model starts off with 99.0% accuracy on the corrupted data (i.e. nearly every sample adversely affected). During unlearning, the base model (i.e. with no added parameters active) accuracy on corrupted data drops to around 10%, which is random chance with 10 classes. This is made possible because at the same time, the corruptions were rerouted into the newly added parameter partition, as shown by the accuracy on the corrupted data for the base model plus the active parameters of the expansion. These results clearly show that the corrupted examples are indeed rerouted, enabling unlearning without the problem of reintroduction that makes prior methods fail.
7 DISCUSSION AND CONCLUSION
We presented a taxonomy for corrupted data unlearning tasks along two dimensions: discovery rate and regularity, and showed that no prior algorithm succeeds across the entire space. We then pro- posed REM, the first unlearning method that performs strongly across this 2D space of unlearning tasks. REM redirects corrupted data to a dedicated model part that is dropped deactivated after unlearning to remove the influence of corrupted data from the model. A limitation of REM is that, as ETD, its masks are binary. Future work could consider softer masking strategies that may allow for better self-organization within 0.,, leading to masks of (even undiscovered) corrupted data having greater overlap with one another. Such better masking strategies may lead to bridging the performance gap with REM (IDEAL) in Fig 3 However, in its current version, REM already makes important strides forward as the first universal method that is strong across our task space. We believe that our 2D framework is an important step forward in better understanding when differ- ent families of unlearning algorithms fail, offering useful tools and vocabulary for explaining their behaviours. We hope that future work expands our framework through identifying other key dimen- sions, builds upon it, for instance by translating its tasks to other modalities, and leverages it for investigating the behaviours of algorithms for other types of unlearning applications.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale train- ing datasets is practical. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 407-425. IEEE, 2024.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7210-7217, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference Learning Representations, 2021. URL hetps: 7/ 7/openrevies ew.
d=Y cbFdNTTy
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intel- ligence, volume 38, pp. 12043-12051, 2024.
Gaurav Rohit Ghosal, Pratyush Maini, and Aditi Raghunathan. Disentangling sequence memo- rization and general capability in large language models. In ICLR 2025 Workshop on Modu- larity for Collaborative, Decentralized, and Continual Deep Learning, 2025. URL https:
//openzeview, www//eeruur/ http://rrmmmm..11000ggggobbuu
Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint arXiv:2201. 066400 2022.
Shashwat Goel, Ameya Prabhu, Philip Torr, Ponnurangam Kumaraguru, and Amartya Sanyal. Cor- rective machine unlearning. Transactions on Machine Learning Research, 2024.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 9304-9312, 2020.
Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like goldfish, don't memorize! mitigating memorization generative 1lms. Advances in Neural Information Processing Systems, 37:24022- 22045, 2024.
Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid false sense of privacy. arXiv preprint arXiv:2403.01228, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regu- larities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 44/26698, 2014.
11

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafilluu. Towards unbounded machine unlearning. Advances neural information processing systems, 36: 36:1957-1987, 2023.
Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? In Proceedings of the 40th International Conference on Machine Learning, pp. 23536-23557, 2023.
Seth Neel, Aaron Roth, and Saeed -allaji-aal Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-533 -57741,,,,, 2023.
Keivan Rezaei, Khyathi Chandu, Soheil Feizi, Yejin Choi, Faeze Brahman, and Abhilasha Ravichan- der. Restor: Knowledge recovery through machine unlearning. 2025. URL https://arx::
org/ 2411. 00204
Stefan Schoepf, Jack Foster, and Alexandra Brintrup. Parameter-tuning-Feet data entry error un- learning with adaptive selective synaptic dampening. arXiv preprinl arXiv:2402. 10098, 2024a.
Stefan Schoepf, Jack Foster, and Alexandra Brintrup. Potion: Towards poison unlearning. Journal
of Data-centric Machine Learning Research, 2024b.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Pro- cessing Systems, 10111887588866 2021.
Ryutaro Tanno, Melanic F Pradier, Aditya Nori, and Yingzhen Li. Repairing neural networks by leaving the right past behind. Advances Neural Information Processing Systems, 35:13132- 13145, 2022.
Reihanch Torkzadehmahani. Reza Nasirigerdeh, Georgios Kaissis, Daniel Rueckert, Gintare Karolina Dziugaite, and Eleni Triantafillou. Improved localized machine unlearn- ing through the lens of memorization. arXiv preprint arXiv:2412.024333 2024.
Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we mak- ing progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint N0Vv:22006000073, 2024.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlcarning. In First Conference on Language Modeling, 2024a. URL
https: http::::///peeeeeve
nwt. forum? fierrmmmdddMXIBXXX jQkmb)
Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent pre-training poisoning of Ilms. arXiv preprint arX1v:2410.13722. 2024b.
Kairan Zhao, Meghdad Kurmanji, George- Octtviann Bärbulescu, Eleni Triantafillou, and Peter Tri- antafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37:12293-12333 2024.
12

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT (OPTIONAL)
We provide all necessary information for reproduction our paper. The codebase of Gooe et al. (2224) for the poison trigger and interclass confusions experiments clearly highlighted in the main paper (Experimental Setup), along with the the random label swap experiments from Maini (2023) and their implementation of ETD. The code for each of these can be found in the original repositories linked in the respective papers (Goel et al. 2022] Maini et al. (023). Paired with our description of REM in Alg. I the detailed loss function eq. [ all necessary hyper- paremeters/model setups in Tab. 3 and the details given in Experimental Setup, the paper is fully reproducible. The GitHub repository linked in the experimental setup paragraph.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 PRIOR WORK MAPPING IN OUR 2D SPACE
The following works are representative for the highlighted areas in Fig. (a).
A Maini al. 2023
B Schoepf et al.
2024b)
C Chundawat et al.
)2033;
Kurmanji et al. 2023
D Goel et al.
2024
Zhao al. 2024
Foster et al.
2024
Chundawat al. (2023); Kurmanji et al. (2023)
Chundawat al.
(2023): Kurmanji et (2033
A.2 HEALING METRIC
The "Healing" results in this paper reported the corrupted data in the training data. We chose accuracy on the training data due to the the observations shown in Fig. [6] Due to the low variance test data in lower regularity settings, the accuracy corrupted data in the training data is significantly more informative. the case of high regularity tasks, there is near linear relationship between behaviour on the train and test data. For example, if the poison trigger is learned in the train data, it will also trigger in the test data and analogously if it is unlearned from the corrupted training data it will not be triggered on unseen test data. In the case of low regularity tasks, there is no connection between the training and the test set. Thus the low variance in the direction of the test accuracy axis. Any impacts of low regularity corruptions on overall model utility are already captured in the "Utility" accuracy that is reported on the test set to avoid falsification by overfitting on the training data.
100
100
100
80
ETD
Healed (train]
False
Healed (train)
60
True
Healed
Healed (test)
Healed (test)
Healed (test)
(a) High Regularity
(b) Medium regularity
(c) Low regularity
Figure 6: We report healing accuracy on the train and test set for all ResNet-9 experiments in the paper for all 10 discovery rates for each unlearning task (three regularity levels) and all UL methods. Results show that healing corrupted data highly correlated between train and test data for corruptions with high regularity while showing little correlation for low regularity manipulations such as Rand. Label Swap. Models that were destroyed during unlearning (model utility below 80%) were removed.
A.3 MODEL AND UL METHOD PARAMETERS
We perform unlearning with a limited hyperparameter sweep as the best values are already known
for high and lower regularity tasks and varying discovery rates from Goel et al. (2024) (poison trigger and interclass confusion tasks). We report the selected hyperparameters in Tab. The ViT architecture and setup for SVHN is adapted from Torkzadehmmhamm et al. (2024). only changing the optimizer to Adam and adapting the learning rate and batch size accordingly. The ResNet-9 setup as well as the poison trigger and interclass confusion tasks are used from Goel et al. (2024).
The random label unlearning problem is adapted from Maini et al. 2023). The REM parameter
14

---

Published as a conference paper at ICLR 2026
Table (Hyper)paramettr overview. The UL learning rate for ResNet-9 the final learning rate of the SGD scheduler during training set by Goel et al. (2024). We use the same 1/5 fraction from starting learning rate to unlearning learning rate for VIT.
Method / Training
Hyperparameter
Values
ResNet-9
Learning rate
0.025
VIT
Learning rate
le-4
All architectures
Epochs
40
ResNet-9
Batch size
512
VIT
Batch size
2048
ViT
Patch Size
VIT
Embedding Dimension
512
VIT
Transformer Depth
4
VIT
Attention Heads
VIT
MLP Hidden Dimension
1536
VIT
Head Dimension
64
VIT
Optimizer
Adam
ResNet-9
Optimizer
SGD
ETD (ResNet)
Layer locations
After each ResNet-9 block
ETD (ViT)
Layer locations
Before/after FF in transformer blocks
All
Seeds
0, 1,
Hardware
GPU
A100 (40GB)
REM, Potion, NPO, Ascent
Threshold Y
0.2
REM
8
SCRUB
a
0.001, 0.01, 0.1,
All UL methods
UL learning rate
0.005 (ResNet-9), 2e-5 (ViT)
All UL methods
Max. UL epochs
10 (25% of training)
is kept at to avoid adding extra complexity to the method by introducing weighting.. Training data splits, choice of which sample IDs to manipulate etc. are taken 1:1 from Goel et al. (2024) to ensure unbiased selection. The hyperparameters for unlearning methods are taken from Goel et al. (2024) and Schoopp'' et al. (2024b). For ETD and REM each mask has 20% the "memorization" partition active during training. The "memorization" partition size is the leftover capacity from "generalization" part to 100% capacity model. For a sensitivity analysis of "generalization" part and mask sizes please refer to Maini al. (2023). From our experiments that seeded with global seed (affecting masks), we observe that REM is not sensitive to the random seed of the mask. Given the three seeds (0. 1. 2), we get the following healing and utility in the ResNet + CIFARIO random label unlearning setting for 50% discovery rate (representative halfway point of discovery): Healing [0.772, 0.774, 0.752.], Utility [0.891, 0.888, 0.896]. Results reported for random label unlearning, as this scenario has the least regularity and highest randomness. Training time on one A100 (40GB) GPU for one seed for the reported results on ResNet-9 and ViT across training types, discovery rates, regularity levels, any UL method hyperparameters is ca. one week depending frequency of logging accuracy results.
Regarding generalisability across architectures, our experimental results show that REM works across architectures with Transformers and ResNet. The implementation in the ViT is agnostic to vision tasks as place the REM expansions before/after the FF layers in each transformer block. This setup is applicable to transformer models across modalities but lies outside the scope of this paper. Ghosal al. (2025) also shows that the ETD approach can be scaled to LLMs for memoriza- tion prevention during training (not an unlearning application), indicating that REM is highly likely succeed at larger scales.
A.4 REM ABLATION
Extending upon the ablations in Table 2 Fig. 7 shows that the barrier step (3.2 in Alg [ adds (i) stability to the results without the performance jumps observed without the barrier step (ii) and leads
15

---

Published as a conference paper at ICLR 2026
to better performance most discovery rates and regularity levels. The few instances where REM without the barrier step outperforms the default REM indicate that adding the barrier to the loss causes some performance loss which is pronounced at lower discovery rates when the loss is noisier due to the smaller sample size.
High-
High
Med.
36.8
41.4
49.2
54.7
59.3 62.6
65.7
69.0
71.0
73
74.3
Med.
36.8
52.0
47.2
50.3
58.6
60
62.1
61.5
64.9
64.8
68.5
Low
45.0
56.0
62.3
63.2
66.5 68.3
71.8
72.9
73.4
75.6
77.2
Low
45.0
57.7
60.3
62.2
64.0
65.
68.1
68.7
70.1
71.3
72.2
0.0
0.1
0.2
0.3
0.4 0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
discovery
discovery
(a) REM
(b) REM without step 3.2
Figure 7: Comparison of REM variants for 50% capacity model on CIFAR1O.
A.5 MODEL CAPACITY AND ADDITIONAL ETD INSIGHTS
In Fig. 8 observe that ETD may be approximated by models with smaller capacity that prevent memorization due to their smaller parameter count. However, in practice it is infcasible to search for a model cpacity that provides the perfect utility memorization mitigation trade-off. Our ex- periments further show that both smaller models and ETD models fail to mitigate high regularity corruptions (see Fig. 8
(c)). This is expected, as high regularity corruptions are informally speaking no different than other concepts the model learns (e.g., concept of stop sign vs concept of poison trigger). We also perform an experiment where we apply our REM masking strategy of assigning each identified corrupted sample the same mask to capture higher regularity corruptions to the model training stage. By giving this combination REM and ETD strategies full knowledge of all cor- rupted samples (information that is not available in practice training time), regularity concepts can be captured effectively as shown in Fig. 9 This not an unlearning method but demonstration of the effectiveness of our novel masking strategy to capture high regularity corruptions. Future work could investigate if leaving already identified corruptions in the training data can be beneficial when adopting this masking strategy from REM create "channel" during training that may capture additional undiscovered corruptions of similar nature.
show that high
A.6 DETAILED RESULTS
We provide detailed results from our experiments in tables and heatmaps that reflect the 2D dimen- sions of our framework from Fig. (a).
16

---

Published as a conference paper at ICLR 2026
Acc on coruptee data
0.75
Acc en corrptted data
0.75
Acc en corrupted deta
0.75
Model capacity
ETD True
50.0
0.50
0.50
0.50
0.25
0.25
0.25
Utiltyy
Utillyy
Ubilty
(a) Low regularity
(b) Medium regularity
(c) High regularity
Figure 8: compare ETD against non-ETD models of the same capacity level, i.e. the parameter count inference, where for ETD, only the generalization partition is active at inference time. The y-axis measures the clean-label accuracy on corrupted data where higher is better. We observe: (i) for low and medium regularity, lower-capacity models fit corrupted data less well compared to higher-capacity models, acting as a type of regularizer against learning corruptions in the first place. However, lower capacity models greatly improves at unlearning the corruptions over its non-ETD counterpart of comparable capacity, which however comes a cost utility, especially at lower capacities. neither capacity restriction nor ETD can move the needle in terms of unlearning corruptions. The drastic utility drop at very low model capacity with ETD likely due to the stark mismatch between the model parameters active during training (for which the loss function optimizes) and inference.
generally associated with lower utility. (ii) in this region, ETD
(iii) For high regularity tasks,
Fulzon Rigger
Band Laved Soop Labbbl
12.2
122
25.5
76.5
76.3
75.5
165
165
75.5
16.5
76.3
15.
0.5
discovary rare
Figure 9: Model trained using the generalization memorization split of ETD but with the masking strategy of REM and given full information about all corrupted samples.
17

---

Published as a conference paper at ICLR 2026
High 13.4
59.3
89.2
89.0
89.4 89.2
89.1
89.4
89.1
89.2
69.3
High
14.2
81.3
81.3
81.2
81.6 81.3
80.8
80.8
80.5
81.6
61.6
Med.
36.8
41.4
49.2
54.7
59.5
62.6
65.7
89.0
71.0
73.1
74.3
Med.
51.8
63.9
65.8
66.
65.8 65.4 65.8 64.7 64.2
63.6
63.6
Low
73.6
0.4 0.5 0.6
0.5 0.6
discovery rate
discovery rate
(a) REM
(b) ETD + REM
Med.
36.8
28.5
31.1 33.3
34.9 36.0 45.2 49.4
50.2
53.0
55.7
Med.
61.8
53.9
55.0
58
58.2 60.5 64.7 65.1
68.2
00.4
70.7
Low
45.0
37.6
39.0 40.5 44.5 45.9 53.1
52.5
56.6
57.9
60.9
Low
74.4
73.4
74.5
75.3
75.7
75.9
76.9
77.2
77.2
77.5
78.1
0.0
0.1
0.2
0.3 0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
discovery
discovery
(c) SCRUB
(d) ETD SCRUB
High
13.4
79.6
80.9
79.7
81.1
81.6
83.6
83.6
84.3
84.8
84.5
High
14.2
71.8
73.6
75.0
76.0
77.0
78.3
77.3
77.7
77.1
76.5
36.88
27.6
74.2
74.4
0.4 0.5
0.6
0.7
0.5
0.6
0.7
discovery rate
discovery rate
(c) Potion
(f) ETD Potion
Hlghh
13.4
83.1
82.0 83.3 82.8 02.9 92.7 02.5 83.2 03.4 82.9
High
14.2
72.4
71.1
70.9
73.0 72.8 69.6
71.3 70.6 69.1
70.7
Med.
36.8
27.3
31.0 29.1 29.4 29.8 32.6 13.1 23.8 32.5 25.4
Med.
51.8
49.3
50.7 51.2
54.9 54.8
51.7 51.1
51.8
51.5 52.2
Low
45.0
55.3
53.7 56 56.8 55 56.1 56.9
57.0 57.8
Low
74.4
74.4
74.4
74.4
74.4 74.4 74.4 74.4 74.4 74.4 74.4
0.4 0.5
0.6
0.4
0.5
0.6
discovery rate
discovery rate
(g) Ascent
(h) ETD Ascent
13.4
85.6
14.2
33.7
36.1
33.9 30.8 18.9 21.6
51.2
54.9
54.8
51.7
51.1
Low
45.0
54.4
54.6
56.2
57.3 57.2
56.0
57.2
57.1
57.0
52.6
Low
74.4
73.6
73.2 73.1
73.6
73.4
71.9
72.7
72.2
72.3
70.8
0.0
0.1
0.2
0.3
0.4 0.5
0.6
0.7
0.0
0.9
1.0
0.0
0.1
0.2 0.3
0.4
0.5
0.6
0.7
0.8
0.9
discovery
discovery
(i) NPO
(i) ETD + NPO
High
13.4
14.4
14.3 13.4 16.1 15.1 16.3 17.4
19.6
24.7
89.2
High
14.2
14.2
14.2 14.2 14.2 14.2 14.2 14.2 14.2
14.2 14.2
Med.
36.8
41.0
44.4 52.0 51.6 51.4 57.5 60.7 67.1 71.0
77.1
Med.
61.8
61.6
61.8 8 61.8 61.8 61.8 61.8 61.8 61.6 8 61.8
discovery rate
discovery rate
(k) Retrained
(1) ETD
Figure 10: Healed x Utility heatmaps for ResNet-9 with CIFARIO and a 50% capacity model unless otherwise specified. Left of dashed line is no unlearning (i.e. trained model before unlearning).
18

---

Published as a conference paper at ICLR 2026
High
70.8
71.9
70.4 69.5
70.3
70.0
71.2
68.7
High
69.3
69.7
69.3
Med.
71.0
73.1
72.5
73.4
73.4 74.1
72.7 73.6
74.2
73
73.2
Med.
72.5
72.6
73.7
72
72.9
72.4
72.5
72.2
74.1
73.6
73.6
Low
74.8
75.7
75.8
76.4
76.4
76.1
76.1
76.4
76.3
76.1
76.1
Low
74.7
75.7
75.6
76.5
76.1
75.6
75.6
76.
76.3
76.0
76.3
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.2
0.3
0.4
0.5
0.6
0.7
discovery
discovery
(a) REM
(b) ETD + REM
10.4
9.9
9.9
74.8
74.7
74.9
75.5
Low
74.8
77.2
71.2
77.3
77.5
77.5
77.1
77.5
77.1
77.5
77.5
Low
74.7
77.5
77.2
77.3
77.8
77.8
77.8
77.7
77.6
77.5
77.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
discovery rate
discovery rate
(c) SCRUB
(d) ETD SCRUB
High-
72
57.2
54.5
55.0
55.4 53.9 56.5 54.8
55.3
55.5
57.8
High
7.0
56
55.8
56.5
54.9 53.7 56.1 56.3 55.4
55.9
57.5
Med.
71.0
71.0
71.0
71.0 71. 71.0 71.0 71.0
71.0
Med.
72.5
72.5
72.5
72.5
72.5
72.5 72.5 72.5 72.5
72.5
72.5
discovery rate
discovery rate
(e) Ascent
(I) ETD Ascent
High
55.9
55.7
57.0
Med.
71.0
68.2
68.7 67.8
67.7 67.7
67.3 67.3
67.2
67.4
67.3
Med.
72.5
69.2
70.1
69.3
69.8
69.1
68.9
69.1
69.1
68.6
60.3
Low
74.8
74.2
74.4
74.5
74.3
74.3
74.5
74.7
74.5
74.8
74.9
Low
74.7
73.9
74.1
74.4
74.2
74.5
74.1
74.3
74.3
74.6
74.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
discovery
discovery
(g) NPO
(h) ETD + NPO
High
7.2
7.4
8.9
9.0
9.8
9.2
96
10.5
54.3
75.7
High
7.0
7.0
7.0
7.0
7.0
7.0
7.0
Low
75.5
76.3
76.7
76.4
75.6
76.1
Low
74.7
74.7
74.7
74.7
0.0
0.3
0.4
0.5
0.6
0.7
0.0
0.9
0.0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
discovery rate
discovery rate
Retrained
i ETD
Figure 11: Healed x Utility heatmaps for VIT with SVHN and 50% capacity model unless other- wise specified. Left of dashed line no unlcarning (i.c. trained model before unlcarning).
19

---

Published as a conference paper at ICLR 2026
Table 4: Aggregate results of methods on CIFARIO with ResNet-9 (50% capacity each) and 1000 corrupted samples with 10 discovery rate levels (10%-100%) and regularities. Presented in de- scending order of Utility x Healed. Error reflects +1 SEM.
Capacity
ETD
UL type
Healed
Utility
Utility Healed
50
REM
81.16 1.62
90.54 0.15
73.40 1.43
50
REM
83.26 + 0.92
88.05 + 0.18
73.19 + 0.72
50
NPO
77.50 = 1.53
86.99 0.24
67.10 1.17
50
Ascent
76.59 = 1.44
86.40 0.35
65.82 + 1.08
50
SCRUB
66.95 =
89.45
5
100
REM
6
91.33
59.65 2.99
50
BadT
66.24 1.89
0.11
58.32 1.63
50
NPO
64.84 2.91
87.59
56.40
50
Ascent
63.98 2.89
86.97
55.50
50
CF
60.24 3.07
8918
5.71
50
Potion
3.66
51.30
50
No UL
56.97
8.10
2071
50
Retrained
53.61 2.73
9
2.47
100
NPO
54.54 3.87
0.36
47.20
50
CF
49.08 + 2.61
0.15
44.51 2.36
100
Ascent
51.98 3.69
85.85
44.20 3.09
50
SCRUB
47.09 2.58
90.71
42.69
50
BadT
46.17 0.84
0..13
41.60
100
Retrained
44.13 2.90
91.46
40.41 2.66
100
Potion
46.79 4.27
5..78
37.1344.11
50
Potion
3.300
36.16 3.62
100
CF
38.82 2.59
+
35.53 2.37
S0
No UL
35.33 1.66
89.94
31.72 1.46
100
BadT
31.18 = 1.63
0.16
28.37 1 1.47
100
SCRUB
26.97 2.05
91.55
24.63 1.86
100
No UI.
15.24 + 0.60
90.87 + 0.15
13.78 + 0.52
20

---

Published as a conference paper at ICLR 2026
Table Aggregate results of methods on SVHN with ViT (50% capacity each) and 1000 corrupted samples with 10 discovery rate levels (10%-1000 100%) and 3 regularities. Presented in descending order of Utility Healed. Values are to the best method for each discovery rate regularity pairing (best by Utility Healed). Potion OOM and thus not reported. Error reflects 1 SEM. No UL and Ascent (best performing prior method) with 100% capacity are reported for reference to the 50% capacity models to show that the observations from ResNet experiments on capacity levels are applicable too.
Capacity
ETD
UL type
Healed
Utility
Utility Healed
50
REM
84.67 0.38
86.53 0.03
73.27 0.32
50
REM
84.17 0.42
86.49 + 0.03
72.80 + 0.36
50
Ascent
81.09 0.74
83.19 0.39
67.70 0.90
50
Ascent
80.28 0.74
83.35 = 0.38
67.14 = 0.89
100
Ascent
79.96 = 1.16
82.99 - 0.78
66.56 + 1.47
50
NPO
79.56 1 0.71
83.08 = 0.38
66.31 0.86
50
NPO
78.65 = 0.78
83.18 0.36
50
Retrained
65.33
86.26
56.36
50
SCRUB
63.36 1:66:35 3.70
87.32 0.02
55.29 3.22
50
SCRUB
63.03 3.71
87.33 + 0.02
55.03 5 3.24
50
CF
62.32 3.70
87.06 0.03
54.27 3.22
50
CF
3.70
+7.3
54.07 3.22
50
BadT
61.20 3.65
85.87 + 0.02
52.50 3.13
50
BadT
61.01 3.59
2 0.02
52,40 3.08
50
No UL
59.94
$ 0.02
51.41 + 3.33
50
No UL
59.44 3.83
85.92 + 0.02
51.02 + 3.29
100
No UL
59.30 2.43
85.83 + 0.56
50.94 2.92
Table 6: Aggregate results of methods on Imagenette with ResNet (50% capacity each. no ETD pretraining) and 100 corrupted samples with 10 discovery rate levels (10%-100%) and regularities. Presented in descending order of Utility x Healed. Values are to the best method for each discovery rate regularity pairing (best by Utility Healed). Error reflects +1 SEM.
UL type
Healed
Utility
Utility x Healed
REM
76.03 2.61
80.32 + 0.28
60.98 2.02
SCRUB
67.63 = 3.63
80.66 + 0.17
54.52 = 2.89
Retrained
65.93 3.04
78.17 0.27
51.49 = 2.32
Potion
56.40 +
2.54
42.61 = -3.83
BadT
47.90 3.18
79.64 0.31
38.09 = 2.50
Ascent
51.37 2.66
75.38
38.05 1.66
No UL
31.00 8.08
79.73 + 1.16
24.69 = 6.38
A.7 ADDITIONAL EXPERIMENTS WITH DIFFERENT DATA, CORRUPTION COUNTS, AND CAPACITY LEVELS
We train the same ResNet as for CIFARIO with the same learning rate etc. for the same 40 epochs on Imagenette (see Tab. 6 7 but with 100 corruptions (to add different size for variety). The new experiments again show REM as the top performing method, even with 90/10 capacity split (Tab. 7. The gap between methods is smaller due only 100 corruptions having lower influence via reintroduction of corruptions in partial discovery settings (i.e. methods fail later in the high regularity setting). Tables and show the same experiments as the main body of the paper but with ETD/REM masks that are 1/4 the size of the main body experiments (20% 5%) to show that results hold across different sizes and do not vary greatly.
21

---

Published as a conference paper at ICLR 2026
Table 7: Aggregate results of methods on Imagenette with ResNet (90% capacity each, with ETD pretraining) and 100 corrupted samples with discovery rate levels (10%-100%) and regularities. Presented in descending order of Utility Healed. Values to the best method for each discovery rate regularity pairing (best by Utility x Healed). Error reflects +1 SEM.
UL type
Healed
Utility
Utility x Healed
REM
77.33 2.72
79.51 + 0,21
61.41 = 2.09
SCRUB
68.83 + 3.90
79.96 + 0.19
55.02 3.07
Ascent
06.97
75.17 0.73
50.21 1.44
Retrained
64.67 2.93
7.25
50.14 2.26
Potion
69.90 1.64
46.28 = 3.16
BadT
55.57 2.49
78.59
43.62 1.92
No UL
45.33 + 8.25
78.50 0.44
35.53 = 6.30
Table 8: Aggregate results of methods using smaller masks (5% active 20% in main experiments) on CIFARIO 10 with ResNet-9 (50% capacity each) and 1000 corrupted samples with discovery rate levels (10% 100%) and regularities. Presented in descending order of Utility Healed. Error reflects +1 SEM.
UL type
Hcaled
Utility
Utility x Healed
REM
85.37
1.77
89.56 0.26
76.38 + 1.48
Potion
78.54 + 2.45
85.84
67.68 + 2.41
Ascent
75.72 = 3.01
88.75
66.81 = 2.38
NPO
75.88 + 3.25
=
66.33 + 2.54
BadT
64.64 2.74
89.28
57.57 + 2.32
Retrained
59.86 + 4.94
88.79
53.21
+: 4.39
SCRUB
59.04 4.92
89.99 + 0.25
53.02
4.36
No UL
51.87 + 19.89
89.47 = 0.93
46.31 17.54
Table 9: Aggregate results of methods using smaller masks (5% active 20% in main experiments) on SVHN with VIT (50% capacity each) and 1000 corrupted samples with 10 discovery rate levels (10%-100%) and regularities. Presented in descending order of Utility Healed. Error reflects
+ SEM.
UL type
Healed
Utility
Utility x Healed
REM
81.19 + 1.16
85.38 + 0.10
69.33 + 1.01
Ascent
77.47 + 1.80
81.41 :
63.49 1 2.03
SCRUB
63.26 6.28
86.21 0.09
54.61 = 5.43
BadT
61.38 = 5.69
0.11
51.61 = 4.80
No UL
59.90 24.29
8..40
50.64 20.55
NPO
59.58 + 6.42
84.40
50.37 + 5.43
Retrained
58.81 + 4.91
76.32 + 0.24
44.98 + 3.75
22

---

Published as a conference paper at ICLR 2026
A.8 COMPUTE TIME AND MEMORY
We report unlearning time as relative numbers compared to initial training for the CIFARIO runs with 50% capacity model (average runtimes). Initial training 100%, Bad Teacher 24.7%, SCRUB 14.0%, REM 10.7%, Potion 9.1 1., NPO 0.2%, Ascent 0.2%. A key consideration in our benchmark that no prior method, not even retraining from scratch, can address the 2D space of unlearning tasks presented. This is different to privacy focused unlearning, where retraining from scratch is the gold standard. Therefore, being more efficient otherwise retraining is better. Given that this is not the case for corrupted data, time is not essential until multiple methods are able to address the 2D space (REM fast nonetheless compared to training from scratch, SCRUB, etc. and only misses out to methods that do not perform any repair steps). Regarding memory requirements, for REM this depends on the size of fog to keep the expanded model in memory. Some methods such Potion have higher peak memory usage due to expensive parameter importance computations (see OOM problem with ViT), while others are more efficient due to no oz as in Ascent, harder to compare due multi-model student-teacher setups as in SCRUB and Bad Teacher.
necessity in privacy focused unlearning
A.9 LIMITATIONS
We highlight the key limitations and assumptions in the main paper. On the unlearning side this is the assumption of having access to the full retain set at unlearning time as is common in the literature. We leave studies more restrictive settings to future work. The main limitation of REM that harms performance is its masking strategy as highlighted in the paper. We also show the potential that can be unlocked by improved masking with REM (IDEAL) that uses information that is not available in practice to reflect perfect masking. Due to imperfect masking, REM will not perform to its maximum potential in practice, which we call out and provide inspiration for future work to address this. We also note that the removal step can be improved by using future advancements in unlearning methods that do not rely on a retain set to replace our implementation of NPO for this step. Depending on the used method for removal, the limitations of the chosen method will be inherited by REM. We show that REM is robust across tasks, models, optimizers and datasets. Prior UL literature has shown that unlearning methods show stable results across varying dataset sizes in high regularity tasks Schoept et al. (2024b) as well as lower regularity tasks of the same nature (e.g. vision classifiers). REM's s runtime is dependent on the chosen number of epochs and comparable to an epoch of normal training as step 2 of REM uses the much smaller forget set and is negligible in comparison to the backpropagation step on the full training data.
23


Reviews:
[{"review_id": "DorODz5C8m", "reviewer": "Reviewer_cRxx", "summary": "This paper studies machine unlearning for corrupted data in vision classifiers. Existing unlearning methods tend to be task-specific and fail when the corruption pattern changes. To diagnose these failures, the authors propose a 2-D taxonomy for unlearning tasks along two axes: Discovery Rate and Statistical Regularity. Through extensive experiments, the authors show that all prior unlearning algorithms succeed only in narrow regions of this 2D space, and fail catastrophically outside their specialized slices.To address this, the paper introduces REM (Redirection for Erasing Memory), redirecting all corrupted data into newly added neurons during unlearning, then drops these neurons so that all corruption effects are removed from the final model.", "questions": "(1)The main conceptual contribution of REM is the claim that corrupted examples can be explicitly routed into a newly added parameter partition. However, the paper provides no network-level analysis verifying that this mechanism actually occurs.\n\n(2)Masking scheme lacks theoretical guarantees and may cause pathway overlap. The redirection mechanism relies on a mask assignment: all corrupted samples share one mask and each clean sample receives a random mask. However, the paper does not justify why this ensures correct sample separation.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)The authors introduce 2D taxonomy to identify statistical regularity as a missing dimension in unlearning research.\n\n(2)The authors proposed REM to redirect corrupted data into dedicated parameters and drop them, inspired by the existing method ETD.\n\n(3)REM is the only method that avoids collapse across both dimensions, compared with baseline methods.", "weaknesses": "(1)EM doubles the model size and is not viable for real-world models. From the perspective of efficiency, the core design of REM expands the model by adding a full set of new parameters. This may be feasible for small CIFAR-scale networks, but it is not realistic for large scale models. Even the VIT structure meets the OOM problem. The paper does not discuss scalability, memory cost, or efficiency, which makes REM difficult to use in real-world unlearning scenarios.\n\n(2)While the empirical results indicate that REM achieves strong performance across discovery levels and regularities, the paper does not provide any network-level evidence confirming that the proposed mechanism, i.e., redirecting corrupted information into θ₀₂ while keeping θ₀₁ clean, actually occurs inside the model.\n\n(3)Incomplete reporting of ViT metrics. In Table 1, the authors report full results (Healed, Utility, and Utility×Healed) for ResNet-9, but only report the aggregated metric (Utility×Healed) for ViT. The omission of the individual Healed and Utility values for ViT makes the evaluation incomplete and prevents readers from assessing the unlearning–utility trade-off on a second architecture.\n\n(4)Questionable OOM explanation for Potion–ViT experiments. The paper states that the Potion–ViT experiment resulted in Out-of-Memory on an A100 40GB GPU. This justification is not entirely convincing: Potion is not significantly more memory-intensive than several other baselines (e.g., SCRUB, BadT, NPO), all of which were successfully evaluated on ViT under the same setup. Moreover, techniques such as gradient checkpointing, reduced batch size, or accumulation are typically sufficient to avoid OOM on CIFAR-10–scale ViT models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbOsXG4SSP", "reviewer": "Reviewer_PYtW", "summary": "This paper introduces a method for unlearning of mislabelled data in the context of classification tasks. Authors first provide a problem definition and a analyse pros and cons of pre-existing works under a taxonomy covering two axes: levels of discovery of corrupted data and corruption patterns. Following this analysis, they propose a two-step unlearning method that borrows ideas from anti-memorisation technology to redirect corrupted information to a new set of dedicated weights. The approach shows more robustness across different scenarios than other methods.", "questions": "-\tIn general, I think this is an interesting contribution to the field of unlearning. The most interesting part of this work is the unlearning taxonomy and analysis of pre-existing work. I would recommend revising presentation in intro/abstract to facilitate reader understanding. \n-\tWhy is NPO not discussed in section 4? It is a crucial part of the methodology in the following section. It would be great to have an explanation of how this approach works conceptually, as it is currently not discussed at all in the paper. \n-\tCould authors clarify what they mean when mentioning 50% and 100% capacity models? Is it just smaller models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The introduced taxonomy, analysis different forms of unlearning tasks in the context of classification problems is a nice framing of the problem. The analysis of a set of unlearning methods, including their strengths and reasons why they fail in certain scenarios was well written and helpful to a non expert in unlearning. Authors provide sensible and well justified arguments. \n\nThe proposed method heavily relies on prior work, but makes intelligent use of ideas developed in those work, which are designed for other problems, and adapts it to the problem at hand. The method is further designed in a way that builds on the analysis provided in the taxonomy section, making the paper more coherent as a whole.", "weaknesses": "-\tOne weakness refers to the inconsistent presentation quality of the paper, certain sections (such as section 4) are much better presented than others (such as section 5). The abstract and introduction in particular try to be too general, making it more difficult to understand the purpose and setting of the paper until section 2. Similarly, the paper mentioning a “universal unlearning method” yields expectations that the proposed work is not restricted to the classification problem.  Certain terms are discussed before being introduced, such as healing in the related works section. \nFigures could also be improved, particularly figure 1 which requires accessing the appendix to be readable. In general, there are too many references to content in the appendix that is needed to read the main paper properly. Please refrain from including experiments in the main paper referring to results reported only in the supplementary materials. The main paper should be self-contained. Ablations are an important part of evaluating new methodology and results should be in the main paper.\n-\tCertain aspects of the methodology are confusing notably how the second fine-tuning stage prevents re-introducing corrupted information in the model. The claim that “undiscovered corrupted examples may be partially caught due to their masks partially overlapping” seems to be more wishful thinking than a rigorous scientific hypothesis. \n-\tIn general, the methodological novelty is somewhat limited, being closer to an orchestration of pre-existing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RklNZeJlyx", "reviewer": "Reviewer_GrjL", "summary": "This paper focuses on a particular machine unlearning setting, where the data is corrupted and the user only partially knows the corrupted data. In other words, some corrupted data still remain in the retain set. The authors propose a method to handle this setting, which can potentially handle cases with a high fraction of unknown corrupted data, as well as corruptions with high regularity. The method is based on Example-Tied Dropout (ETD), which introduces new parameters, and introduce random dropout/masking that is fixed for each example, such that the example-specific knowledge is encoded in these parameters (which can then be removed). The proposed method is largely similar to ETD, although there are certain differences. Results are reported on CIFAR10 with ResNet-9 and ViT with SVHN.", "questions": "The details of the NPO optimization method is unclear, and how it is applied to this classification is also not clear to me. Also, since the NPO optimization is to remove the impact of the forget data from the general parameters, can this be replaced by a more traditional unlearning method?\n\nI am not familiar with the ETD method. How is it that so many parameters can be dropped out at test time, yet the performance is not severely affected? Is there some intuition for this?\n\nIt is unclear how the proposed method can handle cases of corrupted data with high regularity. Is it something to do with the masking strategy?\n\nCould the authors clarify how they measure regularity exactly? Also, how do they vary the regularity in the data?\n\nDid the authors experiment with different ways of adding parameters onto the model? \nAlso, how many parameters were added to the model (which are then removed)?\n\n\nOverall, I currently recommend a borderline accept. However, I am not familiar with several key concepts in this paper as well as ETD, which is strongly related to the proposed method. I will revise my score based on the author’s responses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper has clearly identified a gap in the literature of machine unlearning, where the corrupted data discovery rate and regularity can pose a challenging scenario.\n\nThe proposed method shows good improvements under this corrupted data setting (Table 1).", "weaknesses": "The exact differences from ETD, and why these differences matter, is not very well explained. It seems to me that there are several differences, but I am not able to grasp well these implications, and why these differences are substantial or important.\n\nThe experiment results are compared against methods that are rather outdated. SCRUB and ETD seem to be the latest works in Table 1, but they are both works from 2023. I strongly suggest that the authors compare their works with more recent publications and methods. This would make their claims more convincing and substantial.\n\nMore specific questions are placed in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m3adC2aA2K", "reviewer": "Reviewer_UzFN", "summary": "This paper addresses the challenge posed by the specialization of machine unlearning (UL) methods to particular tasks, which has complicated their systematic comparison and resulted in unpredictable failures in practice. To characterize diverse corrupted data unlearning tasks in vision classifiers, the authors propose a conceptual taxonomy described by two key dimensions: the **discovery rate** (the fraction of corrupted data known at the time of unlearning) and the **statistical regularity** of the corrupted data. Previous State-of-the-Art (SOTA) methods were typically designed for specific regions within this 2D space and are shown to fail predictably and catastrophically outside of these targeted areas. To overcome this limitation and provide a universal solution, the authors introduce **Redirection for Erasing Memory (REM)**. The core feature of REM involves redirecting corrupted data to dedicated neurons that are newly introduced to the model during the unlearning process. Subsequently, these dedicated neurons are discarded or deactivated to suppress the influence of the corrupted data. Experimental results demonstrate that REM is the first unlearning algorithm that performs strongly across the entire spectrum of tasks within this defined 2D space, setting a new SOTA and proving to be a safe choice for practical applications.", "questions": "- Most prior unlearning research treats a fully retrained model as the gold-standard, by the definition of unlearning. However, this paper considers scenarios in which retraining is not a valid gold-standard. Under that premise, can this work still be considered, in a fundamental sense, a contribution to machine unlearning?\n\n- I'm wondering whether the Utility$\\times$Healed metric is essential. Would separate reporting convey the trade-off more transparently in ViT case?\n\n- For LLM unlearnings, what are practical ways to capture or quantify \"statistical regularity\"?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Proposes a clear 2D taxonomy (discovery rate × statistical regularity) that explains why prior unlearning methods succeed only in narrow slices and fail predictably elsewhere. \n\n- Introduces REM, a simple yet novel “redirection” mechanism that routes corrupted data into added neurons and then drops them—yielding strong performance across the entire task space. \n\n- Backs claims with broad, rigorous benchmarks (CIFAR-10/SVHN; ResNet-9/ViT), showing SOTA scores and practical efficiency.", "weaknesses": "- The stated motivation is that existing unlearning methods sometimes fail unpredictably an that they lack a principled understanding of the causes.Yet the paper offers no deep causal analysis; it primarily introduces a taxonomy (but valuable).\n\n- Step 2 states that the original model is updated using $L_{step2} (L_{remove})$. However, in Step 3 the model is updated using $L_{step3}$, where a minus sign is placed in front of $L_{remove}$. Even if Step 2 is indeed the unlearning phase, the current wording is ambiguous.\n\n- The problem setting is well-motivated and realistic. That said, the assumption of full access to the entire training set is difficult to justify in practice. Although the authors defend this design choice as a means to preserve model utility, it would be better that exploring alternatives that operate with only a subset of the training data.\n\n- The loss function notation seems inconsistent, $L$ and $\\mathcal{L}$ are used confusingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)