Mitigating Privacy Risk via Forget Set-Free Unlearning
Authors: Aviraj Newatia
Michael Cooper
Viet Nguyen
Rahul G Krishnan
Venue: ICLR 2026 Poster (2026)
Keywords: machine learning
unlearning
privacy
corrective unlearning
deep learning
risks
approximate unlearning
empirical
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Training machine learning models requires the storage of large datasets, which often contain sensitive or private data.
Storing data is associated with a number of potential risks which increase over time, such as database breaches and malicious adversaries.
Machine unlearning is the study of methods to efficiently remove the influence of training data subsets from previously-trained models.
Existing unlearning methods typically require direct access to the "forget set"---the data to be forgotten-and organisations must retain this data for unlearning rather than deleting it immediately upon request, increasing risks associated with the forget set.
We introduce partially-blind unlearning---utilizing auxiliary information to unlearn without explicit access to the forget set.
We also propose a practical framework Reload, a partially-blind method based on gradient optimization and structured weight sparsification to operationalize partially-blind unlearning.
We show that Reload efficiently unlearns, approximating models retrained from scratch, and outperforms several forget set-dependent approaches. On language models, Reload unlearns entities using <0.025\% of the retain set and <7\% of model weights in <8 minutes on Llama2-7B.
In the corrective case, Reload achieves unlearning even when only 10\% of corrupted data is identified.

Published as conference paper ICLR 2026
MITIGATING PRIVACY RISK VIA FORGET SET-FREE UN- LEARNING
Aviraj Newatia"
Michael Cooper
University of Toronto, Vector Institute Toronto, ON, Canada
University of Toronto, Vector Institute
Toronto, ON, Canada
evinewatia@os toronto.
coopernjecs http///ssttttttoooo Coronto.edu
Viet Nguyen
Rahul G. Krishnan
University of Toronto, Vector Institute Toronto, ON, Canada vietêcs. toronto. edu
University of Toronto, Vector Institute Toronto, ON, Canada
rahulgkēcs wtw.llgkcc..rrrot..eee..e towonto.ootoooooooo
ABSTRACT
Training machine learning models requires the storage of large datasets, which often contain sensitive or private data. Storing data is associated with number of potential risks which increase over time, such as database breaches and malicious adversaries. Machine unlearning is the study of methods to efficiently remove the influence of training data subsets from previously-tranned models. Existing unlearning methods typically require direct access to the "forget set"-the data to be forgotten-and organisations must retain this data for unlearning rather than deleting it immediately upon request, increasing risks associated with the forget set. We introduce partially-blind unlearning selearning-uttlizinng auxiliary information to unlearn without explicit access to the forget set. We also propose a practical framework RELOAD partially -biin method based on gradient optimization and structured weight sparsification to operationalize partially-hlind unlearning. We show that RELOAD efficiently unlearns, approximating models retrained from scratch, and outperforms several forget set-dependent approaches. On language models, RELOAD unlearns entities using <0.025% of the retain set and <7% of model weights in <8 minutes Llama2-7B. In the corrective case, RELOAD achieves unlearning even when only 10% of corrupted data is identified.
1 MOTIVATION
many facets of modern life, individuals consent for institutions to collect and use their personal data. Patients allow their data to be stored in electronic health records, internet surfers allow their browsing behaviour to be used to customize their searches, and citizens respond to public surveys, file their taxes, and register to vote using online government services. Frequently, institutions leverage machine learning (ML) models to derive insights, generate knowledge, or extract value from user data (Shinde & Shah, 2018; Pi, 2021: Sarker, 2021; Rahman al., 2024). However, the act of collecting and storing user's data poses inherent risk to the user. For example, cybercrininals may breach an institution's is data security to commit identity theft (Anderson et al., 2008) leak user data (Kenny, 2018: Zou et al., 2018), or patient records in an electronic health record system may be improperly accessed by curious healthcare workers (Long, 2016). These kind of breaches have the potential cause users financial, clinical, and reputational harm.
"This work was completed while the author was student at the University of Toronto and Vector Institute. A software implementation of our work can be found the project page.

---

Published as a conference paper ICLR 2026
Informally, modern machine learning systems expose the user two types of risk: dataset risk represents the user risk associated with an institution storing user's data, while model risk represents the additional risk the user when their data is used to train a machine learning model. Whereas an ordinary data breach an example of dataset risk, the reconstruction of user data from model weights by malicious actors (Shokri et al., 2017; Haim et al., 2022; Oz et al., 2024) is an example of model risk. If we assume that instances of dataset risk and model risk each occur with some nonnegative rate, Rp, R.M. respectively, we can represent
the instantancous risk borne by an individual at time t as as R(t) = (t( + (t) Since both risk functions are nonnegative, the cumulative data risk and model risk, In (t( dt and So (M) dt, increase with T. This captures the simple intuition that risk compounds: the longer personal data remains stored by an institution embedded within model, the greater user's 's cumulative risk exposure.
"Right to be forgotten" provisions such as the GDPR (Eu- ropean Parliament & Council of the European Union), allow users to demand that their data be deleted from both data stores and machine learning models (via machine learning, Bourtoule et al. (2019))-in theory, eliminating both database and model risk. In practice, because un- learning methods directly employ the "forget set" (Graves al., 2021: Thudi al., 2022: Chundawat et al., 2022; Fan et al., 2023)-the the user data meant to be deleted- institutions must retain the user data to perform unlearn- ing after request for deletion is made. This is typically not a fast process: because instance-wise unlearning is expensive the ever-larger that are becoming common- place, institutions often accumulate deletion requests and process them in batches using conventional forget set- dependent methods (Hu et al., 2023). However, as long as this data is reteined-often for long time, on the order of months the user remains exposed to both R. (t) and RM (f) for all & between when the request for deletion is made and when unlearning has been completed.
Datondt fück
Voctor
Risk
R
Dutaset Risk
Maded Risk
Yeutors
Risk
R (1)
Model Risk
User Cousenta to Add tlvir Data
User Requests Deletion Unkearning Completed
This work develops a procedure for machine unlearning that does not require access to the forget set. Such pro- cedure would allow user data be immediately removed when request for deletion is made, eliminating the con- tinued accumulation of dataset risk (t) after that time (the red region of Figure 1). Our algorithm, RELOAD, combines insights from three families of unlearning and finetuning- -based algorithms-4o implement a procedure that performs unlearning using only minimal aux- iliary information about the forget set, rather than the forget set itself. An organisation using RELOAD would be able to immediately delete user data once a request for deletion made without inhibiting downstream unlearning. Our work makes the following three contributions:
Figure 1: Conventional unlearning algorithms admit a cumulative user risk totalling the sum of the green, blue. and red regions. By allowing user data to be deleted immediately once a request for deletion is made, RELOAD eliminates the user risk associated with the red region.
structured pparsity-based
1. We establish and motivate the purtially-blind unlearning (PBU) setting, capturing the intuition of unlearn- ing without the forget set while leveraging auxiliary information. We characterise it in terms of input requirements, privacy risks, and approximation quality.
2. We introduce the RELOAD algorithm as an efficient algorithm to enable partially-blind unlcarning. Rather than requiring the forget set, RELOAD only requires cached gradients from the final step of training.
2

---

Published as a conference paper ICLR 2026
3. We show that RELOAD consistently outperforms baselines across both standard and corrective unlearning scenarios (Goel et al., 2024). Surprisingly, RELOAD achieves state-of-the ahettht results even when compared algorithms that make use of the forget set. We then extend the RELOAD algorithm perform entity-level unlearning in language models (LMs).
Technology privacy law must be constrained by technological limitations: lawmakers cannot demand the implementation of technically-infeasibe solutions. Although modern technology privacy law like GDPR permits temporary retention of data to facilitate downstream unlearning, we do not believe this provides the strongest privacy outcomes for users. Our work advances the frontier of machine unlearning to lay the technological groundwork for stricter data-deletion timelines under gighttt-bb-forrotte legislation.
2
METHOD
2.1 SETTING AND NOTATION
Let D = {=)]/= with = E 2 represent (i.i.d)) training data from individuals on which an organisation seeks to train a machine learning model. For a class of models M == {Ma : E }} parametrised by a family θ denote the parameters that minimise the empirical risk on D by
0 arg min L(0; D),
lge
where L: 2 - R+ is differentiable, additive risk (loss) function, and denote the trained model as Mg- E M.. For simplicity, we shorten L(0; D) to C(D). Let D forget C D represent the forget set, the subset of training data corresponding to individuals who have requested deletion of their information from the organisation's 's system. Complementarily. define the retain set Dretain == D \ 'D Dorgett- In an ideal world where Mg- is currently in deployment, upon the deletion of D forget, the organisation should deploy new model, model Man, trained on the retain set with
M = arg min L(0: Dretain),
Thus, machine unlearning (MU) aims to transform MM.- into model Ma close to Ma- in some appropriate model-distances (e.g. predictive divergence or weight distance) without costly retraining an entirely new model. A classical unlearning algorithm
function Amu mapping Arr (Mg. D Pretain: 2 forget) to weights 0 E θ such that Mj z Mg- typically by directly using D sorget in the update rule (e.g., targeted gradient steps, reweighting, or pointwise correction). As previously discussed, classical unlearning approaches that require direct access to DD Forget continually compound data subjects' cumulative risk, yet completely eliminating the influence of Doorge from Mg- without any information about the forget set is impossible: we cannot unlearn from nothing. While we must avoid retaining the raw forget set data, we can leverage auxiliary information that was collected during the original training process serving privacy-pres crivacy-preservin proxy that enables effective unlearning while allowing immediate deletion of the sensitive data.
Thus, denote by Zp any auxiliary object derived from training on D that may be retained help perform unlearning without keeping raw examples from D forget. Examples include (but are not limited to) aggregated gradient statisties or feature summaries. We further impose the design desideratum that Ip be chosen so that recovering individual examples in 'D forget from Ip is difficult in practical sense (e.g. low instance-level leakage or computational hardness). We do not require an impossibility claim; rather this a constraint on acceptable choices of Zp.
Definition 1 (Partially-blind unlearning). An unlearning algoritiu APBU operates in the partially- unlearning (PBU) setting if it has access to (i) the trained model Mos, (ii) the retain set Dretaino and (ii)) auxiliary training information Lp. The algorithm outputs Ma = ArBu(Mu (Mg-, Dretainy Ip) such that Mj z Mg-, where 8 is the retraining solution D Pretain-
3

---

Published as a conference paper ICLR 2026
Loes Landscape of C(D)
Loss Landscape of ((Dretain)
Loss Landscape of L(D) - L ((Dretoua)
V.(C()
CID
VACI DD rancs
0. = 0
0 0
Dimeston Atsent
VDL(D)
VDL(D)
VaL(D retain
(1) Cache gradients of the loss the training set, V.C(D) at the end of training.
(2) Compute gradients of the loss on the retain sct, (.D(Dana)))
(3) The difference of these represents the loss landscape fitting the model C(D) but not ((Drain)) As goal la to fit to L(D , we perform one step uscent on this landscupe.
Re-Initialize if:
Loss Lendscape of [(Protts)
Vo, (C(D) ((Dretan))) te Vo. +E
(Potentially far from Ot due to re-initialisation)
Sa
(4) For each parameter 0%, a small ratio of VV (C(1)) -((Prem)) to (V, i indicates that is well-optimized that are not in C(Dreten)) Re-initialize parameters for which this value is small.
points L(D)
(5) Starting these new parameters, optimiize until convergence the landscape of C(D
Figure 2: Overview of the RELOAD algorithm for partially-blind approximate unlearning. RELOAD marries a gradient- based unlearning step modified for the PBU setting (Steps (1) through (3)) with weight saliency-based selective reinitialisation (Step (4)) and subsequent fine-tuning (Step (5)). Because the partially- blind unlearning setting prohibits taking gradients with respect D forget. RELOAD exploits the linearity of differentiation treat
as a proxy for VD forget) at the location in weight space corresponding to 0z. This allows us to apply one gradient ascent step in this direction. Intuitively, this update in Step (3) removes some information about forget from all network weights, while the reinitialisation in Step (4) reinitialises those weights with uniquely strong correspondence to forget (for which single ascent step will fully remove this information). RELOAD achieves state-of-the performance on a collection unlearning tasks, oflen outperforming baselines with direct access to D Forget-
2.2 THE RELOAD ALGORITHM
The RELOAD algorithm is a PBU algorithm leveraging auxiliary object Ip == VoL(D), the gradient of the loss function evaluated on (0", 'D the final epoch of training, in order to eliminate the influence of the forget set from the trained model. RELOAD comprises three stages combining insights from the approaches of Thudi al. (2022), Fan et al. (2023), and Warnecke al. (2023). Figure 2 provides a summary of the method.
1. Ascent (Step (3) in Figure 2). With the trained model Mø-, cached gradients VaC(D), and the retain set Dretain. RELOAD performs single gradient ascent step (Thudi et (2022)) the direction of VoL(D)-
V Vo retain) with learning rate Tpp
8- 8" + (Vo((D) VoL(Drrtain)
Re-initialisation (Step (4) in Figure 2). RELOAD identifies weights responsible for characterizing information about D forget (Fan et al. (2023)) and re-initialises them. Concretely, knowledge values of each weight 0g are computed for the updated weights 8'. For a small c, the knowledge value of 0g
Vo, C(D) Vg, [(Dretain)) +
KV
Vo, C(D) +
4

---

Published as a conference paper ICLR 2026
where a low knowledge value indicates that 0g carries stronger knowledge of forget. Let KV { (KVo, 0g E 8 denote the set of all knowledge values of 6'. For quantile hyperparameter all weights Og with K VV,, I Quaantiee (KV) are re-initialized.. Denote by 0t the weights post selective reinitialization.
3. Finetuning (Step (5) in Figure 2). RELOAD finetunes Mot (Warnecke et al. (2023)) until convergence by minimising ((Dretoin) via iterative gradient-based optimization starting from 0f and obtain 0.
2.3
ALGORIITMIIC INSIGHTS
Partial blindness of VDC(D). RELOAD uses gradients VaL(D) cached upon the completion of the last training epoch as auxiliary training information. While prior work has demonstrated that inputs can be reconstructed from gradient information (Geiping et al., 2020; Zhao et al., 2020; Vero et al., 2023; Wu et al., 2023a; Gao et al., 2021), these reconstruction methods either require knowing batch sizes (unavailable in our setting) or make restrictive assumptions like duplicate labels (Xue et al., 2023). Moreover, reconstructed images are typically unrecognizable with only a small portion showing limited fidelity (Geiping et al., 2020). Thus, cached summed gradients represent valid choice for Ip in the partially- blind setting.
Direction of Movement. The central challenge of partially- blind unlearning is that taking repeated gradients of L(D) forget) is impossible without access to 7) forget- However, from cached gradients of D at the conclusion of model training, VoL(D), we can infer VoL(D) forget) (Appendix A.
V (DL(D forget) VaL(D) - V "eC((Detain)
Therefore, a gradient-bass based descent update in the direction of V (Dogge) moves the model weights such that they better fit to Porget:: because our goal unlearning D Porget. RELOAD instead begins with single gradient ascent update step (in the opposite direction). This informs Step (2 3) in Figure 2.
Targeted Weight Adjustments. Taking a gradient step in this direction insufficient for unlearning for two reasons: we are limited to single step without access to Dj farget, and network modularity theory (Rodriguez et al., 2019) suggests that a small subset of weights contains disproportionate information about D forget- While one ascent step removes some information about D forget across all weights, it cannot fully remove information from the subset most responsible for characterizing the forget set.
We therefore perform selective reinitialisation based weight importance to ensure full removal of in- formation from the subset most responsible for characterizing the forget set. The relative magnitude of Ve, (orget) compared to C(D) represents how much weight 0 is responsible for characterizing 'D forget- A small relative magnitude indicates that 0% is well-optimized to characterise instances in D forget while large relative magnitude indicates that 0₂ poorly characterises these instances. We call this the knowledge value of weight Oz, formally defined as,
KVO,- = C(D +E
Ve, (C(D) [(Dretain)) + (oD)(D) +
(e,
Vos ((D + Vo. L(D)) +
C(D) +E
(1)
where e is a small Laplace smoothing constant. By selectively reinitialising all weights Gg if KVe, S
Quantile, a (KV), where a controls the aggressiveness of re-initialisation selection, we can remove the influence of the weights uniquely responsible for encoding information about D Forget. This informs Step (4) in Figure 2. This thinking extends on lines of work in gradient- based input saliency maps (Smilkov et al., 2017) and saliency unlearning by Fan et al. (2023). We ablate knowledge value formulas in Appendix C.
Due to this tight coupling of components, the produced effect is a large modification to weights which strongly characterise instances in D forget combined with smaller modification the remainder of the weights, enabling wide removal of characterisation of the instances in D forget- We ablate components the RELOAD in Appendix C.2 and C.8 and confirm that these components non-redundant and essential.
5

---

Published as a conference paper ICLR 2026
3 EMPIRICAL RESULTS AND ANALYSIS
Our empirical evaluation has five complementary objectives assess RELOAD's capabilities, (1) Method- ological Inspection empirically verifies each component of RELOAD'S unlearning procedure to ensure well-founded design choices. (2) Classical Unlearning evaluates RELOAD's performance on forgetting individuals' private data points, assessing effectiveness at approximating models trained only on the retain set. (3) Entity Unlearning examines forgetting specific entities concepts in LMs, testing RELOAD's ability to remove knowledge about individuals from the TOFU dataset (Maini et al., 2024). (4) Corrective Unlearning (Goel al., 2024) investigates mitigating training data aberrations when only a subset of affected samples can be identified, focusing on challenging scenarios where fewer than 80% of corrupted samples are identified- reepesienererreeeeettn representnn realistic conditions with incompletely diagnosed data quality issues. (5) Ablations of RELOAD Components provides a complete picture of the algorithm's 's stability across model types, gradient caching techniques, and more.
3.1 METHODOLOGICAL INTROSPECTION
To introspect on RELOAD, we focus on the simplest unlearning task: unlearning class of data from trained model. In this case, we unlearn the class "8" from a ResNet-18 model trained on the SVHN dataset. We further ablate the number of ascent steps, knowledge value formulas, and hyperparameters in Appendix C.
Figure visualises selected feature maps of the ResNet-18 model at different stages of RELOAD and their t-SNE representations (van der Maaten & Hinton, 2008), colored by their predicted label?. The experiment demonstrates the importance of the reinitialisation step (Step (4) in Figure 2), as even after a single ascent step, the model still finds "8" to be the most probable class. Only after the important weights are identified and reinitialised does the model emits a lower-entropy distribution classifying the digit as a "2". This suggests that the primary utility of the ascent step in our algorithm is in amending the representations of D forget in the later layers of the network, while the selective weight reinitialisation step modifies the representations produced by earlier layers. The findings of this experiment provide a degree of empirical confirmation of the intuition presented in Section 2.3.
3.2 CLASSICAL UNLEARNING EXPERIMENTS
Baselines. We compare RELOAD against baseline approaches of GA (Thudi et al., 2022), FT (Warnecke al., 2023), SSD (Foster al., 2023), SCRUB (Kurmanji et al., 2023), CF-k (Goel et al., 2022), EU-k (Goel et al., 2022). SalUn (Fan et al., 2023), and Fisher (Golatkar et al., 2020). FT, CF-k, EU-k, and Fisher are partially-blind algorithms, whereas the others require direct access to Porget. We also present results from ground-truth retraining from scratch, Mgg, to evidence the performance of unlearning algorithms. More details on baselines are provided in Appendix B.
Evaluation. We assess the performance similarity between our learned model and a baseline version of Mg- trained naively from scratch using two key metrics. First, we calculate the forget accuracy difference (AFA, 4), representing the performance gap between our method and the baseline M on 'D Farget- Second, we compute the difference in forget membership inference attack success rates (AFMIA, 4), which quantifies how well the membership inference attack from (Shokri et al., 2017) can identify Forget samples in each model's training data relative to the baseline identification rate on Ma Additional evaluation metrics, experimental details, and hyperparameter configurations are detailed in Appendices B.3, B.7, and B.8 respectively.
RELOAD effectively unlearns both random and correlated samples. We evaluate RELOAD under two regimes: randomly selecting 10% of CIFAR- 100 training samples for D forget, and selecting 100 samples
2 2--NNE visualizations in Figure 3 are of separate, independent run from the left-hand side figure.
6

---

Published as a conference paper ICLR 2026
Coav. Layer
ge Cowv, Conv.
Prediction
o-NE Reprosmiations
Traised ((P)
(Step 1)
18-
(Step 1)
Legend
SVAN
6
Step C(2) 2((Prn))
(Step 3)
(Step 3)
7
8
9
SVEIN Dagk
SVDDN Dip
Ro- lod-untttalioed Salient Weights
(Seep 4)
(Step 4)
3374
(Stop
SVEN
Figure 3: Introspecting selected feature maps of ResNet-18 model when using RELOAD unlearn the class "8". (Left) The feature maps (activations) of the first channel fo the first, third, fifth convolutional layers after the application of each algorithm step in Figure 2. After Step (3), the activations of the model remain largely unchanged. although the logits represent a considerably more uniform distribution over the digits. After Step (4), the activation of the first convolutional layer is largely unchanged-whinn broad fcature detectors (Zeiler & Fergus, a0lh))-although the feature maps change and the predictive distribution contentrates around "2". (Right) t-SNE visualisations of the 5th Conv. Layer a ResNet- 18 trained on the same task a different training run. At Step (1), classes largely cluster into nice separated regions with some overlap. The Ascent Step (3) disrupts the cluster separation, preserving some clustering structure while breaking apart much of the original class boundaries. Reinitialisation (Step (4)) reveals that the model collapses onto label at this stage (on the left '2', the right, "8" though our experience it coincidental that this run happens to concentrate the class we desire unlearn). Finally, Finetuning (Step (5)) recovers much of the original separability, restoring the distinct class clusters seen in the initial t-SN. Importantly, we observe that no samples are predicted to be "g".
is expected, the earlier layers of CNNs tend to correspond
10% Random
100 In- Cl-Class
Entity Unlearning
Algorithm
AFA (1)
AFMIA (1)
AFA (L)
AFMIA (L)
Algorithm
FQ)
MU (t)
GA
18.77 + 2.43
0.21 + 0.06
23.33 + 1.06
0.07 + 0.06
GAA
0.0068
-0.0233
SSD
74.17 + 2.04
0.15 1 0:21
68.67 1.97
0.38 + 0.14
Grad Diff
0.0143
-0.0198
SCRUB
18.85 + 2.39
0,20 0.06
27.55 1.43
0.07 = 0.06
NPO-RT
0.5786
-0.161
CF-k
18.01 + 2.60
0.20 1.20:::00 0.06
21.81 0.88 0.88
0.07 + 0.06
Pref Opt
0.0971
-0.0021
SalUn
13.14 + 2.53
7.39 2.60
12.08 3.13
0.02 + 0.02
ECO (Zero-Out)
0.9900
+0.0000
Fisher
22.99
2.30
7.27
0.04
Original
0.0030
+0.0000
RELOOA
0.30 0.50
0.01 + 0.01
3.44 + 1.46
0.02 + 0.02
RELOAD
0.4046
+0.0748
Table I: Benchmarking RELOAD against baselines in the uncorrelated 10% setting, 100 in-class samples setting, and 1% forgetting entity unlearning setting. Best performances are bolded.
from a single class to assess unlearning of correlated data. Partial results are reported in Table while com- plete results deferred to tables in Appendix B.4. In both settings, RELOAD achieves strong performance across key metrics. For random sample unlearning, RELOAD attains the highest RA while maintaining the lowest AFA, AFE, AFMIA, and FSKL, suggesting superior approximation of Mg- compared to baselines. For correlated sample unlearning, RELOAD achieves the lowest AFMIA and FSKL. and performs compet- itively other metrics, closely approximating My-. While Fisher marginally outperforms RELOAD in the correlated setting, it requires over twice the computational time as retraining, making RELOAD more
7

---

Published as a conference paper ICLR 2026
practical. Methods like CF-k and EU-K achieve higher RA in the correlated setting due to minimal weight updates, but perform poorly on critical unlearning metrics like AFA and AFE. Surprisingly, these results demonstrate that RELOAD's 's superior approximation of M&- enables more effective unlearning than methods that explicitly leverage D forget during the unlearning process We observe that RELOAD outperforms both GA and FT consistently, which form parts the RELOAD algorithm. This supports the utility of how RELOAD is designed, and that each step of the algorithm plays a critical role in unlearning.
3.3 ENTITY UNLEARNING WITH LMS
Baselines and Evaluation. We compare RELOAD against baselines from Maini et al. (2024) on entity unlearning using TOFU's synthetic author biography dataset (Maini al., 2024). We evaluate using forget quality (KS test p-value between unleamed and retrained model distributions) and model utility (performance on retained data and real-world knowledge). Experiments use Phi 1.5 (Li al., 2023) and Llama-2-7BB B-Chat (Touvron et al., 2023) with open-source fine-tuned models (locuslab, 2025; Unlearning, 2025a;b;c).
RELOAD unlearns select entities. this experiment, we task each algorithm with unlearning a subset of the fictitious authors in TOFU from a TOFU fine-tuned model. We observe that RELOAD effectively unlearns when the number of entities to forget is small. As evidenced Table RELOAD outperforms many existing unlearning methods in the 1 1% forgetting case, effectively forgetting entities (g-value >> 0.05) and improving model utility over the Original and Retrained (Retain) models. However, the 5% and 10% forgetting cases, RELOAD fails to repair model utility despite succesful forgetting. We hypothesise that this is due to limits on the size of Drepoir. Due to computational restrictions, the size of Drepaér was restricted (maximum 195 samples) which greatly limited the applicability of RELOAD when [Dprompts| As the 1 % forgetting case is the only case in which (D prrrmpts I [Drepairl this suggests that this bound is a requirement for the effective application of RELOAD for entity unlearning. Similarly due to computational constraints, we reuse results and the reference implementation for experiments from prior work (Liu et al., 2024a). Further experiments are provided in Appendix B.5.
|D (Drepairl=
The experiment on Llama-2-1 2-7BBChat completes in 8 minutes on single RTX6000 GPU, using 7% of weights and <0.025% of retained data, demonstrating RELOAD 's efficiency for small-scale entity unlearning.
3.4 CORRECTIVE UNLEARNING
Baselines and Evaluation. Corrective unlearning (CU) (Goel et al., 2024) considers the case where a portion Dm of has been adversely affected (e.g. mislabeled or poisoned). CU aims to update so as to approximate training on D \ Dnm where only a subset DD farget C D,n has been identified. Existing methods degrade rapidly when y == [D forgell/ / [Dmm < 0.8 and fail under adversarial corruptions Pawelczyk et al. (2025). To gauge performance on CU, we use the corrected accuracy Acc (f), measuring the performance of the unlearned model on the adversely affected data D,.. Full details on prior work, baselines, and evaluation metrics for CU are detailed in Appendices A.5, B.2, and B.3.
100
SSD
Scruly
RewoD
Alcuee
RalT'
Reload
large-scale poisoning
0.1
0.8
10
Idlutiifed Fracioo 7
Figure 4: CU results CIFAR-10 for dataset poisoning with [Pm 100. RELOAD achieves high Acceor across all y E (0, 1.0) whereas base- line methods often struggle with low 7.
RELOAD efficiently corrects trained models. Under adverse effects of manipulations following the baselines outlined in prior work, RELOAD outperforms on Acc at low percentages of data identification (Figure 4) while observing competitive computational efficiency (Table 23) even at only y = 0.1.
Although BadT (Chundawat et al., 2022)
8

---

Published as a conference paper ICLR 2026
outperforms RELOAD in CIFAR- 100 poisoning experiments, it bears much greater computational cost (Table 23). We report further experiments and results in Appendix B.6.
3.5 ABLATION OF RELOAD COMPONENTS
To further characterize the behavior of RELOAD, we conducted an extensive suite of ablations across various architectures and data regimes. These experiments investigate the impact of gradient quantization to mitigate the storage overhead, the stability of the algorithm when applied to Vision Transformers to characterize our method across model types and architectures, performance continual learning and fine-tuning scenarios for real-world practicality, its efficacy on human sensitive datasets for unlearning private data (Newatia et al., 2024), and how RELOAD behaves when only using subsets of Dretain for finetuning.
We defer the reader to Appendix C for detailed analyses and discussions.
4 RELATED WORK
Exact and Approximate Unlearning, Exact unlearning provides formal guarantees for information removal from model weights. Methods include naive retraining (the gold standard (Cao & Yang. 2015; Thudi et al., 2022; Shaik et al., 2024)), SISA (Bourtoule et al., 2019) for accelerated retraining via data partitioning, Certified Data Removal (Guo et al., 2019) using reverse Newton updates, and Certified Graph Unlearning (Chien et al., 2022) leveraging graph topology. Approximate unlearning methods like RELOAD recover retain-set behavior without theoretical guarantees. Approximate unlearning algorithms can be classified into gradient-basse baseed and weight- malhhny-aliency approaches. (i) Gtradient-based approximate unlearning methods perform optimization using both forget and retain sets. Simple approaches apply gradient ascent on forget loss to undo weight updates (Graves et al., 2021; Thudi et al., 2022). Teacher-student methods include Bad Teacher (Chundawat et al., 2022), which distills from models trained on retain data ("good teacher") and randomly initialized on forget data ("bad teacher"), and SCRUB (Kurmanji et al., 2023), where students learn to disobey teachers by maximizing forget loss. Representation based methods include DUCK (Cotogni et al., 2023), driving forget representations toward incorrect centroids, and Boundary Unlearning (Chen et al., 2023) for class-level decision boundary shifts. (ii) Weight saliency-basee approximate unlearning methods target specific weights based on neural modularity (Pfeiffer et al., 2023) and sparsity (Frankle & Carbin, 2018; Chen et al., 2024). SalUn (Fan et al.. 2023) uses gradient thresholds to identify forget-sensitive weights, while SSD (Foster et al., 2023) scales weights using Fisher Information Matrix importance scores without gradient steps.
Partially- Blind Unlearning, Related to Zero-Shot Unlearning Chundawat et al. (2023) (restricted to class unlearning), this setting more realistic and applicable. Methods include Finetuning (FT) (Warnecke et al., 2023) on retain sets, Catastrophically forgetting last & layers (CF-k), Exact-unlearme et al., 2022), and Fisher Forgetting (Golatkar et al., 2020). Both FT and CF-A provide no strong theoretical indication of unlearning while RELOAD provides stronger theoretical indication by selectively reinitialising weights bearing the most knowledge on D forget.
last & (EU-k) (Goel
Unlearning for LMs. Most methods optimization to balance forgetting undesirable sequences while retaining useful ones (Yao et al., 2024; Liu et al., 2024b; He et al., 2025). Some identify and edit sparse weight subsets (Wu et al., 2023b; Ilharco et al., 2023; Belrose et al., 2025) but require large retention datasets or full-model updates (Eldan & Russinovich, 2023). Optimization- free methods have been explored (Liu et al., 2024a). RELOAD offers lightweight alternative requiring minimal data, few updates, and fast convergence.
9

---

Published as a conference paper at ICLR 2026
5 DISCUSSION
RELOAD effectively unlearns arbitrary samples. Despite operating in the partially-blind setting, RELOAD outperforms MU algorithms that enjoy direct access to forget. However, RELOAD trades off runtime and performance in unlearming arbitrary samples of data, requiring the caching of summed gradients over D from the final stop of training, a non-trivoe non-trivial spatial cost. When 2 forget is available, RELOAD does not require this caching the gradients can be computed runtime. In both settings, RELOAD 's method proves to be an empirically effective approximate unlearning method.
RELOAD causes LMs to forget entities. In applications to LMs, RELOAD is able to quickly and cheaply remove knowledge of entities when the number of target entities is S the size of the subset of the retained data used for computing knowledge values. Applications of our algorithm also leads an overall increase in model performance. When this condition isn't 't met, RELOAD fails to achieve both model utility and forget quality. In addition, RELOAD achieves this without needing to modify the inference pipeline of these models (as opposed to baselines such as (Liu et al., 2024a)).
RELOAD corrects data aberrations. In corrective unlearning, RELOAD remains an efficient, performant method in this regime, outperforming existing baselines and serving as a viable approach for all currently explored forms of corrective unlearning. RELOAD demonstrates its ability to unlearn manipulations when < 80% of the manipulated data is identified in all corrective cases, presenting a step up from prior results (Goel et al., 2024). This holds practical value when y is not known in deployment and suggests that our work may contain generalizable insights about about learning to fit arbitrary downstream transformations of data.
RELOAD discards batch statistics. Caution is required for applying RELOAD to models using Batch Nor- malization (e.g., ResNet-18). Since removing the forget set alters the batch statistics (mean/variance) for the retained set, the gradients are technically coupled. However, in practice, we observe that this "approximation mismatch" is negligible. As detailed in Section 3.2, RELOAD achieves competitive performance ResNet- 18, suggesting the method is robustly handles the minor gradient drift introduced by batch- aatch-dependent statistics.
6 CONCLUSION AND LIMITATIONS
Further work on partially-blin algorithms, in particular RELOAD, could focus on behaviour in contrastively trained models. Investigating the effect of a RELOAD -style unlearning latent representations of contrastivell-trainee model is an exciting future direction. Crucially, the focus in such a setting shifts from measuring traditional performance gain or drop with respect to the retrain baseline, assessing the identifiability and similarity of the learned representations between the unlearned model and the retrain baseline. For instance, future work could focus on metrics that quantify how closely the representation space of the unlearned model aligns with that of the retrained model, such as centered kernel aligmment (CKA). It is further important to study whether RELOAD performs well on models with self-supervised pretrining and subsequent finetuning using task-hased decomposable loss such as cross-entropy or if decomposable losses can serve as a surrogate for RELOAD unlearning on models which are not trained with such loss.
By enabling unlearning without direct access to the forget set, RELOAD addresses the fundamental privacy paradox in machine unlearning: that the very data subjects wish to remove must be retained during the unlearning process. This capability allows organisations to immediately delete requested data upon receipt of deletion requests, effectively stopping the cumulation of database-relan related privacy risks rather than perpetuating them through batched processing delays. In doing so, RELOAD represents a meaningful step toward aligning technological capabilities with regulatory mandates for deletion "without undue delay" and the privacy protection goals underlying tigh-to-be-forgotten legislation. While our work demonstrates substantial empirical improvements across multiple unlcarning scenarios, future research may explore theoretical bounds for partially-blind unlearning, broader model classes, and further reducing auxiliary data requirements.
10

---

Published as a conference paper ICLR 2026
ACKNOWLEDGMENTS
The authors would like to thank Anvith Thudi, Patrik Reizinger, Siddharth Arya, and Alexander Capstick for constructive conversations unlearning. The authors would also like to thank Younwoo (Ethan) Choi for insights on contextual fine- soeitunnig for large language models alongside Asic Chen and Chenika Bukes for discussions neural network initialization techniques, unlearning, and weight sparsity. The authors would also like to thank the RGK lab members for general discussions, support, and feedback. RGK is supported by Canada CIFAR AI Chair and Canada Research Chair Tier in Computational Medicine (CRC-2022-00049), This research was supported by an NFRF Special Call NFRFR2022- 00526. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute
This research was enabled in part by support provided by Compute Ontario (ttiss//www.compicontuuio and the Digital Research Alliance of Canada https:/www..aliinncca..co
REPRODUCIBILITY STATEMENT
In efforts to promote reproducibility the authors have provided the source code used to run the experiments.
REFERENCES
Keith B Anderson, Erik Durbin, and Michael A Salinger. Identity theft. Journal of Economic Perspectives, 22(2):171-192, 2008
Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 6679-6687, 2021.
Arthur Asuncion, David Newman, al. Uci machine learning repository, 2007.
Nora Belrose, David Schneider-Joseph. Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form, 2025. URL htwpt: http:////aavvvxa/aaaeeee /2306. 03819.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Chin CChoo, Hengrui Jía, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. 12 2019. URL http://arxiv, org/abs/ 1912. 03817.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium Security and Privacy, pp. 463 480, 2015. doi: 10. 10.1009/P..015.35.
Asic Chen, Ruian Ian Shi, Xiang Gao. Ricardo Baptista, and Rahul G Krishnan. Structured neural networks for density estimation and causal inference. Advances in Neural Information Processing Systems, 36, 2024.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7766-7775, 2023. URL https: / /tpp.ppp..///// www.stissannnnii.....mee org/
CorpusID: :20.3::::27776774000
Eli Chien, Chao Pan, and Olgica Milenkovic. Certified graph unlearning. ArXiv, abs/2206.09140, 2022, URI https: /tppi/ppp..p// semant ieweeatttsccccco cscholl ar orge ID: : 24989011
Younwoo Choi, Muhammad Adil Asif, Ziwen Han, John Willes, and Rahul Krishnan. Teaching llms how to learn with contextual tine-tuning, 2025. URL https: //tp://aaavvv//a//////eaeaawavva org/ abs abs/2503. 09032.
11

---

Published as a conference paper ICLR 2026
Vikram Chundawat, Ayush K Tarun, Murari Mandal, and Mohan S. Kankanhalli, Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. ArXiv, 0bs/2205508009 2022. URL
https:/ //tpa////pp../// semant cecchlacrrrr
Vikram Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlcarning. IEEE Transactions on Information Forensics and Security, 18:2345-2354, 2023. ISSN 1556-6021, doi: 10.1109/tifs.... (009/tiis..2233335555511 URL http:t/pstee..tt.etee 3265506.
1109/TIFSS.... 2023.
Marco Cotogni, Jacopo Bonato, Luigi Sabetta, Francesco Pelosin. and Alessandro Nicolosi. Duck:
Distance-based unlearning via centroid kinematics. ArXiv, abs/2312.02052, 2023. URL https:
/ / /aap.. www..peeaaaaaaeeea.emaaaaaaeppaa
-emannt nt icscholar. org/Corpus : 65666966977
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010. 1929, 2020. URL ht.pp:: : //t://appp /app.. semant cscho wac.ooor....ooooo org/ Ctrg/Corpul roup ID: 225039882
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms, 2023. URL https: / wrwii. org/ abs//bsss. 02238..238...
European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Couneil. URL ht wwt.t:tps//om http::://daaaaaaa www.aaaurrpppaaoco
wwt.oppp.eeeelii wwtweeeeeeeeeeeeeeeeeeeeeeeeeeee wtte///////////eeeeeeeee 679//j.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearming via gradient-based weight saliency in both image classification and generation, 2023.
URL https: //tpxvaaaavv../aa//eaa/aaaaaaaia org/ abs/ 2bs//23 /2300 12508.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through se- lective synaptic dampening. ArXiv, abs/2308.07707, 2023. URL https: //app/appa////// sewantiianiii. acc
orq/CozpusID: 220900355555
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks.
arXiv preprint 18X3:08003333335 2018
Tianxiao Gao, Wu Wei, Zhongbin Cai, Zhun Fan, Shane Xie, Xinmei Wang, and Qiuda Yu. Ci-net: Contextual information for joint semantic segmentation and depth estimation, 2021. URL https:
/ //arxiv. weciiivvorrggeee org/abs.......ooo........eeo...e /tabs///11.. 2217.. 13800.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients - how easy abbs////////// /tabb//2000 22000 1 14053.
is it to break privacy in federated learning?, 2020. URL htttps://arxiie
Robert Geirhos, Jörn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 665-673, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Staristics, volume of Proceedings of Machine Learning Research, pp. 249 256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: http::::////cceeeer//// mlr. press v /////llloe glorot10a html.
Shashwat Goel, Ameya Prabhu, and Ponnurangam Kumaraguru. Evaluating inexact unlearning requires revisiting forgetting. ArXiv, abs/2201. abs/2201.06640, 2022. URL https: / //pp.....////////.// iemaanicsccooare
12

---

Published as a conference paper ICLR 2026
Shashwat Goel, Ameya Prabhu, Philip Torr, Ponnurangam Kumaraguru, and Amartya Sanyal. Corrective machine unlearning. Transactions on Machine Learning Research, 2024. hRLpt httttp:://ppe.ree..eee... JBB..
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020. doi: 10. 1109//cpp446000020200005053 URL, http: /xx doi. dog..og/1...... 1109/CVP342600 0020.00225 00932.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens der Maaten. Certified data removal from machine learning models. ArXiv, abs/1911.03030, 2019. URL https: htpps://aple semanticschooioeo..o
org/CorpussD::078441
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv [cs.LG], June 2022.
Estrid He, Tabinda Sarwar, Ibrahim Khalil, Xun Yi, and Ke Wang. Deep contrastive unlearning for language models, 2025. URL http::tttt://ate .rgg abs 2503. 15503.1400.... 14900.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10.1 /0/110/022.22..1222..
Yuke Hu, Jian Lou, Jiaqi Liu, Wangze Ni, Feng Lin, Zhan Qin, and Kui Ren. ERASER: Machine unleaRning in MLaaS via an inferencE seRving-aware approach. arXiv [cs.CR], November 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023. URL https: ///aa.xxvvvoog. org/ abs 2212. 04089.
Caitlin Kenny. The equifax data breach and the resulting legal recourse. Brook. J. Corp. Fin. & Com. L., 13: 215, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, 6628106.
2014. URL httph::t///ett hta.:pppp//app..emmattcccoooool www.tttiiccccoaaaoooooooooeeeeee
Ron Kohavi. Census Income.
UCI
Machine
Learning Repository, 1996.
DOI:
http:://doi.or/1122442////
Alex Krizhevsky, Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 100 (canadian institute for advanced research). URL http: /wwww/wwwww///////////// cs. toronto.....oooooo edu/-kriz ci sifar.. html.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. In Teirth- Conference on Neural Information Processing Systems, 2023. URL https:
//openreview net / www.ormmmii-oveaaatttat
Yuanzhi Li. Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. URL https:ttp........... /taaaaivvv.//////a/ /ttp//awwww....aaa.a.s.oso 2aabs/2399 22099 054633363
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embedding-cormutee prompts, 2024a. URL https::/arxiv..rr////////// ab2rg//../224....../// 079333
13

---

Published as conference paper ICLR 2026
Sijia Liu. Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models, 2024b. URL Atttsstt:///xiiiiti abs/ 2402. 08787.
locuslab.
tofu ft -apiii900ppii11555 retain90_phhi 5, 2025.
https:
Kurt Long. Do no harm: the insider threat to patient data. Engineering & Technology Reference, (2016), 2016.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter, Tofu: task of fictitious unlearning for Ilss, 2024. URL https: : / aaaiiiiiv wtgg/bs////a///e 06122,
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential leamning problem. Psychology of Learning and Motivation, 24:109-165, 165, 1989. URL https: //ppi.appp..////
semanticscholaz: www.oo////ooooooo Coopgg//orpssD: 61019113.
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011. URL https: / /api. semanticscholes wwtp:orrr/oooooooooeeeeoo 16852518.
Aviraj Newatia, Michael Cooper, and Rahul Krishnan. Unlearning tabular data without a "forget set", In NeurIPS 2024 Third Table Representation Learning Workshop, 2024. URL net/ wiw.ummmidDDNNiiiiiie
Yakir Oz, Gilad Yehudai, Gal Vardi, Itai Antebi, Michal Irani, and Niv Haim. Reconstructing training data from real world models trained with transfer learning. arXiv [cs.LG], July 2024.
Martin Pawelczyk, Jimmy Z. Di, Yiwei Lu, Ayush Sekhari, Gautam Kamath, and Seth Neel. Machine unlcarning fails to remove data poisoning attacks, 2025. URL https: //axaiv///xvvvv.// org/absg /206. 17216.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Ponti. Modular deep learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. httttttts////eeeeee www../ee/eeeee forum? id=z9EkXEvxta. Survey Certification.
Yulu Pi. Machine learning in governments: Benefits, challenges and future directions. JeDEM-eJoural of eDemocracy and Open Government, 13(1):203-213 2021.
Anichur Rahman, Tanoy Debnath, Dipanjali Kundu, Md Saikat Islam Khan, Airin Afroj Aishi, Sadia Sazzad, Mohammad Sayduzzaman, and Shahab S Band. Machine learning and deep learning-bast based approach in smart healtheare: Recent advances, applications, challenges and opportumities. AIMS Public Health, 11()): 58, 2024.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? 2018. https://arxiv. org/ abs/1806..... 00451.
Nathaniel Rodriguez, E. Izquierdo, and Yong- Yeol Ahn. Optimal modularity and memory capacity of neural reservoirs 1, 2019.
Iqbal H Sarker. Machine learning: Algorithms, real-world applications and research directions. SN computer science, (3)):60, 2021.
14

---

Published as a conference paper ICLR 2026
Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape of machine unlearning: A comprehensive survey and taxonomy, 2024.
Pramila P Shinde and Seema Shah. A review of machine learning and deep learning upplications. In 2018 Fourth international conference on computing communication control and automation IntooaationICCCUSS (ICCUBEA), pp- 1-6. IEEE, 2018
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18, Los Alamitos, CA. USA, may 2017. IEEE Computer Society. doi: 10.1109/SP.2017.41 URL https://doi.
Leeecomputersocietr www.ciiitttttgg.e..oo.ee wwg/10....... 11090......11111 00110//...21....0. 2017.41.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: remov- ing noise by adding noise. ArXiv,
semant 2017. URL L
arg/Corpus ID: D 11 695878.
A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&amppP)) pp. 303-319, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society, doi: 10. 109//ErrSSP5384422222 00027. URL https::/ doi. 00027.
www.eiitttt.ogg.looooooe 11t.1111/////////33444444. 20222....22.... 2022.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonpara- metric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30 (117:198--1970, 2008.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edumov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models, 2023. URL
.rgg abs 22007 09288.
Open Unlearning.
https: https: https::/hugginggacee.o open aeeen-uneeamiig/tofuu.. hf 7 2025a
tofu 1-llaaa-----chatthfffaaaa
2- www.cchathhhfcc
Open Unlearning. Ttaeraaaaaaa-a-------hhahheaeaaa 2025b.
nofu_llama-22 hhh._llama---hbbhhhhhhheeeee
Open Unlearning tofu hoit_llamaa---bbccatthfeeeee
https://huggingaace
open-unlearning/toone
- 9tw.:ii9erettiiin,e__e 2025c.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning
Research, ((86):2579-260) 2008. URL html.
Mark Vero, Mislav Balunović, Dimitar I. Dimitrov, and Martin Vechev. Tableak: Tabular data leakage in federated learning, 2023. URL httts:::////att org/ abs//2al..002/////// 01785.
15

---

Published as conference paper ICLR 2026
Phil Wang, Zack Ankner, Yonghye Kwon, murufeng, Minh-Long Luu O. Srikumar Sastry, Steven Walton, roydenwa, Ali Hassani, Artem Lukin, Baraa sameeh, JacobLinCool Jason Chou, Jonathan Tow, Kale Kundert, Loe Truong, Ryan Russell, SOUMYADIP MAL, umbertov, Zhengzhong Tu, Aditya Mishra, Andrés, and shabie. lncidrains/vit-pytorcch. https: /tts://gittt /ttp//ttttu..t/ggo/ wwtpc////waaaascccccaaaacacaaaaa p-t--yytt.....h
nov 22 2025. URL
Alexander Warnecke, Lukas Pirch, Christian Wressnegger), and Konrad Rieck. Machine unlearning of features and labels. In Proceedings 2023 Network and Distributed System Security Symposium, NDSS 2023. Internet Society, 2023. doi: (0.147722ndss.2223.230008 URL http: / wtt..oooo.....1122222220 ndss 2023.022222 23087.
Ruihan Wu, Xiangyu Chen, Chuan Guo, and Kilian Q. Weinberger. Learning invert: Simple adaptive attacks for gradient inversion in federated learning, 2023a. URL httpp::/ httttp/ss//axxx.ooo////b.///// /2100 220.22101..00.....11111111111000 10880.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models, 2023b. URL htttps://aaaa...
Dongyun Xue, Haomiao Yang, Mengyu Ge, Jingwei Li, Guowen Xu, and Hongwei Li. Fast generation-based gradient leakage attacks against highly compressed gradients. In IEEE INFOCOM 2023 IEFE Conference on Computer Communications, pp. 1-10, 2023. doi: 10. 10L1000NNOOCO03399922200 202.33999999222222999999
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning, 2024. URL https:// arxiv. orgg abs / 2.3 0 1863.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part I 13, pp. 818-833 Springer, 2014.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients, 2020. URL /ttps://arxiv... awwiv. www.rwwwwww...iieoeoooo. abs/ 2001. 02610.
Yixin Zou, Abraham H Mhaidli, Austin McCall, and Florian Schaub. '''ve got nothing to lose": Consumers' risk perceptions and protective actions after the equifax data breach. In Fourteenth Symposium on Usable Privacy and security (soups 2018), pp. 197-216, 2018.
16

---

Published as conference paper ICLR 2026
A FORMAL TREATMENT AND GRADIENT DERIVATIONS
A.1 THE RELOAD ALGORITHM
Algorithm 1 The RELOAD Algorithm for Partially-Bie -aind Unlearning
I: Input: Ma,, cached VaL(D), Dretain
weights: 7p: priming step learning rate, e: noise weight, a: reset proportion
Output: Trained model approximating M
4:
5: procedure RELOAD(Mp', VoL(D; M(or)), Dretasa)
+ th (V)((D) FoL(Dretain)
b. Step (2- (Fig.
7: KV + Po ((D)) Vo, C(Pretain)
Step (3) (Fig. 2)
No. 2(2])+
ONE0
for Os 0' do
if QUANTILER a then 0/ (NITIAIZZZE end if end for
10:
Step (4) (Fig. 2)
11:
12:
13:
Train M(er) convergence on Dretain
Step (Fig.
14:
procedure
Our RELOAD algorithm (Fig. 2) contains the following steps based on the intuition presented in Section 2.3.
(1) Cache the gradients VoL(D) at the end of training.
(2) Compute V (DD(Drettin))
(3) Perform one step of gradient ascent in the direction of VoL(D) VoL(D (.rraani))))
(4) Reinitialize all weights 0g that are smaller than the a- a-QUANTTILE of knowledge values.
(5) fine-tune until convergence on L(D
A formal description of this algorithm is shown in Algorithm 1.
For entity unlearning in language models, the RELOAD algorithm requires modifications. The design of RELOAD for LMs is outlined Appendix ??.
17

---

Published as a conference paper ICLR 2026
A.2 GRADIENT INFORMATION AND DERIVATION
Information contained in gradients. RELOAD relies on information about D contained within the cached gradients, raising the question how it behaves in the modern setting where networks are trained to convergence. We first observe that [V - 0 does not imply ||V[(S)]le + 0 for S C D. This means that even if the summed- caaned gradients are all approximately zero, Step (3) may still induce non-trivial weight updates. For the same reason, the numerator (from Equation 1) in Step (4) is non-zero.
However, as | + O, the denominator proaches This does not influence the behaviour of RELOAD because this scaling by & applies uniformly to all knowledge values, and a is a quantile of the empirical distribution of the knowledge values. As such, a constant scaling factor applied to the knowledge values will not affect which weights are re-initialized. Figure 5 shows a smoothed empirical distribution of knowledge values demonstrating the spread of values com- puted for different weights across ResNet- 18 network.
Equation ?? ap-
544
RELOAD relies on equating VeL(D forget) to VeL(D)- VD We provide full derivation below following from the lin- earity of differentiation and is used to justify the RELOAD algorithm in the partially-blind setting of classical machine For the purpose of corrective unlearning (with replacement), addi- tional arguments are required and presented in Appendix A.5.
12
30
20
12
Figure 5: (Smoothed) empirical dis- tribution of knowledge values com- puted over the weights of ResNet-18 trained SVHN.
unlearning and corrective machine unlearning.
Gradient Derivation. Recall that D = { with 2; E 2. Let 2; = (x;, y1) represent an input-output pair in the dataset. Then, let is Mo(x;) represent the model's prediction z;. Subsequently,
V (orgee))
Σ
(oL((111)))))) =
Σ (0 (D. DrettDnn
VeL((z,,
(2)
where the second equality follows from Dretain =DiDjorget forget- Equivalently,
Σ
VoL((I, 3i). - ((z.n)) reetiin (x.(((x)))))...
(3)
(ED
Σ
VoL(( U:),
Σ
VoL((Iry))),00
(4)
(DD
(PaAK)
VoL(D) - V (DL(aretain..
(5)
A.3 THE RELOAD ALGORITHM FOR LANGUAGE MODELS
RELOAD on LMs leverages insights from investigations into language models (Choi et al., 2025) to produce richer gradients. This is necessary as LMs are prohibitively-lare for the standard RELOAD procedure which requires Dretain and cached gradients. By reducing the requirements RELOAD for LMs to D Dprompte: we enable practical partially-blin unlearning LMs without modifying the inference pipeline.
this setting, we assume access to a model trained on dataset D. Additionally, we assume access to a dataset of prompts inquiring about 'D) forget, D prompts (eg. 'Who is Harry Potter?' if if forget contained texts on Harry Potter). Finally, we assume access to a small sample of the retain set Drepair S Dretain- We also select subset of model layers unlearn from, D Dlayers, with total parameters @ dxelected- All gradients and
18

---

Published as conference paper ICLR 2026
computations are performed solely on these layers, enabling parameter-effiiint optimization and unlearning. Given that modern language models are trained on very large datasets and contain billions of parameters, it is crucial to be able perform unlearning with small datasets in a parameter-cffiiiet manner (operating on a small subset of model weights).
This procedure contains the following steps.
(1) Collect the outputs of the model on D Proompss in D Dutppus.
(2) Embed each element of D Poutputs in contextual fine tuning (Choi et al., 2025) to create D Pembedded (see Appendix A.4).
(3) Collect the outputs of the model on D Dembedded D Dembedded ootpptts-
(4) Collect language-modeet modelling gradients (eeparr) but do not update parameters.
(5) Collect anngaage-moelllin gradients CDD (Ve((eebeedee embedded cmleddeed_outputa) oottputs), ) and update parameters (one step of gradient ascent).
(6) Reinitialize all parameters 0g rselected that are smaller than the e--UANTTIL of knowledge values.
(7) fine-tune until convergence on C(D ((Prepasr).
A formal description is also shown (Algorithm 2).
It is important to note that in this setting, RELOAD does not require storing gradients, reducing space utilization and overhead. However, due to the requirement of prompts in D Proompss it offers weaker privacy guarantees (ie. knowing which concept is being unlearned).
Algorithm 2 The RELOAD Algorithm for LMs
Input: Mg., Pprompte- Drepair
Parameters: Tpi priming step learning rate, e: noise parameter, o: reset proportion
3: Output: Trained model approximating M
4:
procedure RELOAD- DELM. - Ppromptes Drepair)
Doutputs )(a-) ((aos(Dpromptt) (Dprompts)
Dembedded t- CFF((Douppua) Dembedied_outpure & Mirr)(Demaa (eembedded)
8:
KV IV,, L(P A(Pembedde ooueputt 18 ((Drepei>)
Step (3) (Fig. b> Step (2- 3) (Fig. 2)
O2 Elleetected
10:
O'ntectted <- 600 O'stected + Va([((Deteenn embedddd ouspute)))
11:
for du E d'stlcttdd do
12:
if QUANTILE (KV,,) then 0% (NITIALIZE() if
13:
Step (4) (Fig.
14:
15:
end for
16:
Train M(ry convergence on Drepoir
Step (5) (Fig. 2)
17:
end procedure
A.4 CONTEXTUAL TONE-TUNING FOR UNLEARNING FROM LANGUAGE MODELS
Choi et al. (2025) outline that embedding prompt with their method improves the model's ability to capture underlying functional relationships. In addition, this method of fine-tuning LM's has been shown to lead improved learning. Therefore. parameter updates informed through this method are more informed suggesting that the gradients produced may be more informed about parameter knowledge themselves.
19

---

Published as conference paper ICLR 2026
We find that using this method of contextual embedding produce gradients for RELOAD leads to better identification of knowledgeable parameters about D prompts- This wholly informs the functionality of our unlearning algorithm, leading quick and effective forgetting from LMs.
Given a prompt and model M, we extract the knowledge contained within the LM by performing a forward pass on the prompt to obtain (x). (ar) is a textual representation of what the LM knows about the concept prompted in v. We then insert M(z) into the [content) field of the prompt presented below which we pass to a black-box LM (in our experiments we use the Anthropic API to access claude-33 1-rdde-----aaee 3----haku---- the most knowledge about a topic from an LM. We then perform a forward pass on to yield M (y) on which we collect language modeling gradients.
4122). This yields contextual fine-tuning prompt, y. aimed extracting
The following is the fine-tuning prompt used for black-box LMs for unlearning:
Based on the TARGET CONCEPT:
Generate a concise "contextual prempt" that will enhance learning effectiveness and draw out all relevant knowledge.
The prompt should:
1. Follow the style [select one learning theory approach: In-Depth Explorattion/Refleeiii
and Synthesis/Tosus Key Concepts/Contextual Understaanding/Crttiia Ana Analysis Rand Tearning]
Explicitly identify:
The fundamental concepts that must be understood
Key zelationships between important elements
Critical facts that requi re focus for mastery
How these elements connect to and are relevant for reasoning or application
3. Be formatted
directive that encourages active engagement with the material (approximatelyy
3-5 sentences)
4. Frame learning in way that facilitates long-term retention, practical application, and maximizes extracting knowledge from the learner. TARGET CONCEPT: (cortent)
Your contextual prompt should help the learner not just nemorize information but develop deeper, more applicable understanding of the concept.
A.5 CORRECTIVE UNLEARNING AND GRADIENT DERIVATION
A.5.1 CORRECTIVE UNLEARNING
Goel et al. (2024) introduce the setting of corrective unlearning in which subset of the training data Dn, C DD is adversely manipulated. This could include mislabeling, backdoors, and data poisoning. The corrective unlearning case studies the ability of unlearning algorithms to unlearn the adverse effects produced by the presence of these abberations the training dataset when a sample them are identified (Dforget C Dm)). This extends machine unlearning beyond privacy-related deletions. Interestingly, in this setting, a naively- retrained model is not the gold-standard, as manipulated data may remain in Dretain unknown to the practitioner. Existing methods degrade rapidly when 7 proree < 0.8, and fail under adversarial [am
corruptions or large-scale poisoning (Pawelczyk al., 2025).
Following prior work, let Dm S 'D denote the subset of training points that are adversely affected (mislabeled, corrupted, or poisoned), Only a portion of these may be identified as the forget set. Corrective unlearning aims to obtain Mj R Mgg. Therefore, we write
[Djorget] [Dml
D Dooret C DD
7
E [0, 1],
5 e arg min L(0,D Dm)), Ago
so is the fraction of adverse instances that are identified. The difficulty of corrective unlearning increases as y decreases because D Dretain = D DDDoorgee forget not D D m-
20

---

Published as a conference paper ICLR 2026
A.5.2 CORRECTIVE UNLEARNING (WITH REPLACEMENT)
addition to studying the corrective unlearning case, we provide weaker extension of the corrective setting. Our expansion to the corrective unlearning setting is applicable when the identified samples in 2) Pooget are additionally transformed, corrected, or amended and included in Dretain. In this setting. let f + denote transformation, and write z = f(z)). Then, Dretain represents the result of applying f item-wise K elements of DD and applying the identity transform to the remaining - elements, as
D Daatain = (=/)e=1,...,,K U {=i)i=K-I,....
(6)
This an extension of the corrective unlearning problem, we wish to "unlearn" the influence of ii on our original model, and "relearn" the influence of {zī}i= K. This additional setting encompasses the following data transformations, among others:
where x represents 1. Covariate Correction: Dretain {2/ (2 b= U (z))i=Kt1.....
corrected version of the features TG. and indices K + -1...., N....,NN, correspond to those with erroneous covariates (e.g., data was corrupted during jnllectioonprr-ppoeesnnng))
2. Label Correction: Dretain = {=1-- = (ait, (kk,y/))e y/)ki,,..... U i K,,,,, where 1/ represents a corrected version of the label y/s and indices K + L,,.., N correspond to those that were originally mis-labelled during annotation.
3. Backdoor Removal: Dratain = {=/- (ale, bi)).= U {(a;, u1) j:) where xi represents a version of the features 2i lacking the injected backdoor pattern, and indices K + 1,... N correspond to those that were originally transformed with backdoor during processing, Models trained with backdoors in the training set learn shortcuts (Geirhos et al., 2020), which can be exploited to induce misclassification.
For simplicity, we use 8 to denote the complement of the set or dataset S.
In classical and regular corrective unlearning, our goal is to obtain a gradient in the direction of D forget for RELOAD. The corrective unlearning (with replacement) case more general: the goal to obtain VoL n Dr. De a gradient pointing towards the empirical minimum of the loss on elements that are uniquely contained in D and not Dretains and CDL((D' Dretain), a gradient pointing away from the empirical minimum of the loss on elements uniquely contained in D, Dretain but not in D. This is general abstraction of the difference in gradients between dataset and a subset of that dataset, to the difference in gradients between two datasets. Unlearning represents the special case of this framework in which n Deetain = D forget and De n Dretain = 0. In the corrective unlearning (with replacement) setting, the desired gradient is also V L(D) VoL(D ); for which we provide justification and a derivation below. This validates Step (2 3) in RELOAD for this problem.
We now outline the derivation of the gradients informing the RELOAD algorithm in the context of corrective unlearning (with replacement). Recall that in this case, Dretain # D - 'Doorget, which invalidates the justification outlined in Section 2.3. Below, we provide derivation which justifies the same choice of gradient for the corrective unlearning (with replacement) setting.
In the setting of corrective unlearning (with replacement), we construct these sets.
Dn (2+ (II, 31) i D Dretain {z] (4'r, ))== KI.. and D Dretain {2+ (Ie, Me) kem..K
(7)
(8)
21

---

Published as a conference paper ICLR 2026
The gradient then formulates
VoL(( 9c)
Σ VaL((c Vo v:),
(9)
(
(N..0:)ED
(( +
((r' OkaKkt....
-1 (2.U))
Vol((ri v:).
(E.u)) [VL[((1;;
(10)
DhDretan
= VoL((( v.). - Σ VoL((ar vc).
- ()) + Vol((( v.).
(11)
(EwN))DD Pretain
(srA)))Pretom retaoin
VoLl((ri:ii) Σ
u))
(12)
(s..)L)D VoL(D) (oL(Dratni))
(13)
In this work, we consider the Corrective Unlearning (with replacement) cases that correspond to the classical corrective unlearning scenarios outlined by Goel al. (2024). We recreate the experimental settings exactly, except the samples in D) forget are corrected, and included in Dretain-
B EXPERIMENTAL DETAILS AND RESULTS
B.1 BASELINES FOR CLASSICAL UNLEARNING
Gradient ascent (GA) (Thudi et al., 2022). GA operates by taking several steps of gradient ascent on forget thereby removing the trained model from a loss minimum on forget. This approach is not partially-bling
Fine-Tuning (FT) (Warnecke et al., 2023). FT leverages the concept of ceaeroppiccalyyforgetteie (Mc- Closkey & Cohen, 1989) to unlearn forget by fine-tuning Dretain- This approach is partially-bling
Selective Synaptic Dampening (SSD) (Foster al., 2023). SSD studies the amount of information about Dorget contained within weights using approximation of the Fisher Information Matrix. Proportional to each weight's 'importance', SSD scales the weight value to induce forgetting. This approach is not partially-blind.
Scalable Remembering and Unlearning Bound (SCRUB) (Kurmanji et al., 2023). SCRUB alternates optimising between distilling away from the original model D forget and towards the original model on Dretain. Notably, the second distillation loss combined with a task-specific loss (eg. cross-entropy for classification).
Catastrophically Forgetting the last k-layers (CF-k) (Goel et al., 2022). CF-k leverages the concept of ceastrophicallly-fogetiiii (McCloskey & Cohen, 1989) by freezing all but the last k layers of the model and performing fine-tuning on Dretain. This approach is partially-blind
Exact Unlearning the last k-layers (EU-k) (Goel et al., 2022). BU-k reinitialises the weights of the last k layers, freezes the rest, and fine-tunes on Dretoin. This approach is partially-blind.
Salience Unlearning (SalUn) (Fan et al., 2023). SalUn is framework in which important weights Dorget are identified and all but those are frozen for optimisation updates. Authors report the greatest improvement when combined with Random Labelling (RL) (Golatkar al., 2020). RL assigns random labels to instances in forget and then fine-tunes on this data. This approach is not partially-blind.
Fisher Forgetting (Fisher) (Golatkar al., 2020). Fisher leverages the Fisher Information Matrix over Dorget to perform Fisher-regularised weight update to the model. This approach is not partially-blic blind.
22

---

Published as conference paper ICLR 2026
B.2 BASELINES FOR CORRECTIVE UNLEARNING
Baselines are taken directly from Goel et al. (2024). We repeat their details below.
Retrain without Deletion Set (RewoD). RewoD represents a naively retrained model on Dretain D Dorget. Notably in this setting, some affected samples of Dn,, may still be in Dretain-
Catastrophically Forgetting all layers/Finetuning (CF) (Goel al., 2022; Warnecke et al., 2023). CF leverages the concept of catastrohiiclly-ffrreeingg (McCloskey & Cohen, 1989) by performing fine-tuning on Dretain.
Selective Synaptic Dampening (SSD) (Foster et al., 2023). SSD studies the amount of information about Jorget contained within weights using an approximation of the Fisher Information Matrix. Proportional to each weight's 'importance', SSD scales the weight value induce forgetting.
Knowledge Distillation from a Bad Teacher (BadT) (Chundawat et al., 2022). BadT uses a combined distillation approach by learning from randomly initialised network on Jorget and the original model on Dretain.
Scalable Remembering and Unlearning Bound (SCRUB) (Kurmanji et al., 2023). SCRUB alternates optimising between distilling away from the original model on D forget and towards the original model on Dretain. Notably, the second distillation loss is combined with task-specific loss (eg. cross-entropy for classification).
B.3 EVALUATION METRICS FOR MACHINE UNLEARNING
We present the evaluation metrics used in our experiments comparing the RELOAD algorithm with baselines. An up arrow f indicates that the higher the better, while down arrow + indicates that the lower the better.
One of the goals of unlearning to produce a model that a close approximation of the naively retrained one. FSKI. and RSKI. are evaluation metrics which quantify the dissimilarity between the outputs of the unlearned model and the retrained model on the same data. AFA, AFE, AFMIA are comparison metrics to benchmark the difference in performance between the unlearned model and the retrained model on the same data. Similar behaviour on D Porget implies that the unlearned model is indistinguishable from a retrained one. Unlearning is only useful procedure if it is cheap and yields useful model. RA measures the utility of the unlearned model, and Cost measures how expensive the unlearning procedure is, relative to retraining from scratch.
23

---

Published as conference paper ICLR 2026
Statistic
Abbr.
Description
Accuracy on
Model accuracy on Dretain. unlearning, higher accuracy indicates that the unlearning pracess has not negatively impacted the model's performance on the retained data.
Dretain (t)
Diff. in
AFA
The change in accuracy the forget between the current model and MM".. smaller difference, approaching the accuracy the retrained model, indicates that the unlearning method has heen more effective in "forgetting" the forget set.
Accuracy on
Forget (
Dift. in Error
AFE
The reduction in error the forget set between the current model and M A smaller difference, approaching the error of the retrained model, signifies that the unlearning method has been more effective "forgetting"
D forget (L)
forget set.
Diff. MIA
AFMIA
Difference success rate a membership inference attack (MIA) on the forget set between the current model and M In this work, use the attack from (Shokri et al., 2017) implemented in the repository for (Kurmanji al., 2023). A success approaching that of the retrained model implies the forgotten data indistinguishable to an MIA on in-distribution data that the model was not trained on.
Success Rate on D forget (L)
Symmetric KL--Diveggence on Dretain (1)
RSKL
Symmetric KL--Divergence hetween the logits of the current model and those of AM This metric averaged all instances in the Pretain- A lower symmetric KI. divergence indicates an unlearning method behaves similarly on the Dretain a model retrained from scratch without the forget set.
Symmetric KI--Divergence D forget (L)
FSKI.
The Symmetric -iv-eiveeeence hetween the logits of the current model those M This metric averaged over all instances in the D Porget- A lower symmetric unlearning method that behaves similarly on the D forget to a model retrained from scratch without the forget set.
KL divergence indicates
Cost (L)
Cost
Ratio the runtime the unlearning method to runtime of retraining a baseline model from scratch without the forget set. A lower cost indicates a more computation- ally efficient method.
Table 2: Evaluation Statisties for Unlearning.
Corrective Unlearning (with and without replacement) Evaluation Metries
Statistic
Abbr.
Description
Retain Accuracy (t)
AcCretain
Model accuracy on held out test set (D(teat) corrective unlcarning. higher accuracy indicates that the corrective unlcarning process has correctly adapted the model to its new training set.
same distribution Dretavn-
Corrected Accuracy (t)
Acceum
Model accuracy on the adversely affected data D.n. In the case of backdoor attacks or noisy corrective unlearning, a higher value indicates the relearned model correctly has lost its reliance on the backdoor pattern. In label correction setting, the desirable value the percentage of samples that did have their labels flipped (in our experiments, 90%).
Cost (L)
Cost
Ratio of runtime of the corrective unlearning method to runtime of retraining a haseline model from scratch without the forget set. A lower cost indicates a more computationally efficient method.
Table 3: Evaluation Statistics for Corrective Unlearning.
24

---

Published as a conference paper ICLR 2026
Acrre and AcCoor are metrics introduced in Goel et al. (2024) measure the effectiveness of corrective unlearning algorithm. Cost is the same as above, to compare the unlearning algorithm to the expense of full training.
B.4 CLASSICAL UNLEARNING RESULTS
We present full set of experiments exploring the effectiveness of the RELOAD algorithm the classical unlearning task of item unlearning. This suite of experiments include unlearning randomly- samples and correlated samples. Across these categories select 10% and 30% of training data samples for random- sample unlearning, and 100 data points from single class for correlated-sample unlearning. These cases are explored across 3 datasets (CIFARIO, CIFARIO0, and SVHN) and models (ResNet-18 and VGGI6-BN).
As previously discussed, RELOAD operates in a partially-blind setting. This means that within the results presented below, RELOAD performs this unlearning without access D Doogett
RELOAD unlearns randomly-selecterr samples. We randomly assign 10% of CIFAR-100 training samples Dj orget, showcase how well each method can unlearn arbitrary training samples from a ResNet-18 and report our results in Table 8. RELOAD achieves the highest RA, while maintaining the lowest AFA, AFE, AFMIA, and FSKL of all approaches. This suggests that RELOAD successfully approximates Mg- better than the baselines. That FT achieves lower RSKL than RELOAD is hardly surprising, as RSKL measures dissimilarity logits on Dretain and FT adjusts a converged model Mg- to fit a subset of its original task. Similarly, the computational cost of RELOAD, though similar to many baselines, is considerably greater than either SSD or GA.
RELOAD efficiently unlearns correlated samples. We randomly assign 100 samples from a single class of the training data to D forget. to evaluate how well each method can unlearn arbitrary but related training samples and report our results in Table 15. RELOAD achieves the lowest AFMIA and FSKL of all approaches and very close to the lowest AFA, AFE, and RSKL of all approaches, suggesting that RELOAD closely approximates Ma in this setting. RELOAD is marginally outperformed by Fisher in these settings, but is far more feasible, as Fisher requires over twice as much time as retraining. Although RELOAD achieves an RA competitive with that of most baselines, naive gradient ascent, CF-k, and EU-k yield a marginally higher RA. This can be attributed to the small number of unlearning samples; optimizing to maximize the loss on these samples does not provide a strong enough gradient update. CF-A and EU-k both make few weight updates Mø-, which leads to a high RA but poor performance on unlearning metrics like AFA and AFE.
Further experimental results on random 10% and random 30% forgetting are provided in the tables below (Appendix B.4.1, B.4.2). Results on 100 correlated-samee sample unlearning are provided in Appendix B.4.3.
25

---

Published as conference paper ICLR 2026
B.4.1
RANDOM 10% FORGETTING
(1)
APA (4)
APE
AFMIA (4)
(4)
(1)
GA
20.21
3.86=o.68
0.21=20.07
0.04
0.00
0.06-o.02
0.66
FT
98.2420.21
1..5
0.16gp-03
0.03
0.2710
0.05
0.4844.04
SSD
20.02 90
75.65_26.
1.8820.62
0.01
0.0110.00
8.30 8.30_3.31
7.83
SCRUB
98.41
3.80
0.21 0.21=0007
0.0440..22
0.02+0.00
0.06-6.02
0.60++.04
CF-k
98. 25.28.20.22
3.81-0.71
0.21=20.07
0.05-2o.02
0.21+0000
0.00=o.02
0.50-+0.04
FU-A
20.21
3.83
0.21
0.05
0.20:20.00
0.07=0.02
0.56
SalUn
99. 0..00
3,08
0. 0.2622002
0.01
0.1610
0.06=0.0z
0.6010.04
Fisher
99. 00.11
3.83-0.44
0.07
0.0220.00
1.83
0.07=0.02
0.56
RELOAD (OURS)
99.4944..10
1.83g0.8
0.05 20.04
0.00.0000
0.261
0.12_6.01
0.53
Retrained (Baseline)
99.9920.01
94.4020.72
0.23
2.50
Table 10% Random Forgetting on CIEAR-IO (ResNet-18)
t: the goal to have as high a value as possible, A"; the value in the table the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have low difference, J: the goal is to have low of value possible. The hottom row presents the absolute value of M(a-) each metric. For any metric with A, the raw value is instead reported. Rows for AFA (J). AFE (1), and AFMIA (4) present the absolute difference the value of the corresponding method on this metric to the value of M(r- on the metrie. These results show that RELOAD outperforms all the baselines on RA, AFE, AFMIA by large margins. RELOAD performs competitively on the AFA, FSKI, and RSKL. metrics, but is outperformed by FT. RELOAD incurs a higher computational cost than other baselines other than FT.
Meihod FT
RA (f)
AFA (4)
AFE
AFMIA (1)
Cost (4)
RSKL (L)
FSKL
98.38
4.40_0.41
0.18220.2
0.05
0.00
0.06__0.22
0.66
08.24+0.21
4.33
0.18-=0.02
0.04:4o.01
0.26+0.02
0.05
0.48-+0.04
SSD
20.02
75. 75.41
1.8922.62
0.0220.03
0.0110.00
8,30
7.83
SCRUB
98.41.0.20
4.47
0.19200.2
0.05 01
0.021 60
0.06_e.o0
0.65
CF-A
98. 0..23
0.23
4.47
0.19=0.02
0.05
0.17+0.01
0.06=0.02
0.55
EU-k
98.31
4.48-0.40
0.19-0.02
0.06+0..1
0.17+0.01
0.07-6.02
0.56++0.04
SalUn
99.86
1.98_0.48
0.09=20-22
0.04g0.01
0.17:2000
0.06=0.02
0.55
Fisher
99.61
0.15-0.08
0.00.go.00
0.01
2.17+0.04
0.07-0.02
0.56+0.04
RELOAD (OURS)
99. 0.
0.08
0.0120000
0.00
0.12g0.01
0.05
0.19
Retrained (Baseline)
99.99+0000
95.16-0050
0.20-0002
0.50+0.00
Table 10% Random Forgetting SVHN (ResNet-18)
t: the goal to have as high a value as possible, A the value in the table the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have low difference, J: the goal is to have low of a value possible. The bottom row presents the absolute value of M(a-) each metric. For any metric with A, the raw value is instead reported. Rows for AFA (1), AFE (1), and AFMIA (4) present the absolute difference in the value of the corresponding method on this metrie to the value of M(rr) on the metrie. These results show that RELOAD outperforms the basclines on RA, AFA, AFE, AFMIA, FSKL, and RSKL by large margins. RELOAD performs competitively on the Cost, but incurs a higher computational cost than other baselines other than FT, CF-k, BU-k.
26

---

Published as a conference paper ICLR 2026
Method
(t)
AFA (4)
AFE
AFMIA (4)
Cost (4)
RSKL (L)
FSKL
GA
98.40
4.43
2.21
0.03
0.00
0.06_0.02
1..0
FT
28.30
4.49
0.22=0002
0.03
0.24+0.03
0.05
0.4922004
SSD
22.88
70. 70.46-22.04
1.80
0.01
0.00
7.99-3.52
7.00+3.06
SCRUB
98.4320.22
4.50
0.22.20.02
0.03.20.01
0.02g0.00
0.06-0.02
0.66
CF-k
98.3410.24
4.51
0.22
0.04g0.01
0.21
0.06_0.02
0.56
EU-k
00.31
4.51=0.42
0.22=2.02
0.04.go.01
0.21++.05
0.06-0.02
0.56+20.05
SalUn
99.9410.02
3.88
0.13-g0.0
0.04g.0.01
0,151
0.06_0.02
0.55
Fisher
99.55g0.15
0.04
0.00
0.00
1.4610.03
0.06=0.02
0.56.0.00
RELDAD (OURS)
99.50=0.11
0.65
0.04.20.44
0.00
0.26+0.10
0.12-6.01
0.53
Retrained (Baseline)
99. 9..0..0
95.08-0.51
0.24=2002
0 0.50
Table 10% Random Forgetting on SVHN (VGG16-BN)
t: the goal is to have as high of a value as possible, A": the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, J: the goal is to have low of value possible. The bottom presents the absolute value of M(a-> cach metrie. For any metric with A, the raw value instead reported. Rows for AFA (J), AFE (4), and AFMIA (4) present the absolute difference in the value the corresponding method on this metric the value of M(g-> on the metrie. These results show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins aside from Fisher. However, Fisher incurs substantially higher Cost, making it far less efficient than retraining from scratch. Therefore, Fisher is impractical, and RELOAD demonstrates the best practicality competitively on RSKL and FSKL but outperformed by FT. RELOAD also incurs higher computational cost than the other baselines.
unlearning mechanism. RELOAD performs
Method FT SSD SCRUB
RA (f)
AFA (4)
AFE
AFMIA (4)
Cost (J)
RSKI. (4)
FSKI.
98.41
26.40_1.18
1.6420.07
0.14 oa
0.00
0.06
0.66g0-00
98.27
12.05=1.81
1.16=20.07
0.08
0.254:.03
0.06
0.50
22. 22.86
61.38_15.72
20.7
0.02
0.00
8,01_3.03
7.57 13.77
98.43_0.23
26.62_1.10
1.66
0.14
0.02g 60
0.06
0.66
CF-A BU-A
98.30
26. 26.26
1.68
0.15+0..2
0.27+0.04
0.06
0.50
98.35
26.16
=0.7
0.15
0.27:t0.4
0.08
0.57
RELOAD (OURS)
99.51 99.51
3.37
0.40
0.02
0.24
0.11=0.01
0.51 0.51
Retrained (Baseline)
97.8010.33
68.20
1.8220.00
0.50
Table 7: 10% Random Forgetting CIFAR-100IVGG1--BN
t: the goal is have as high of value as possible, A+: the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, 4: the goal is to have as low of a value as possible. The bottom row presents the absolute value of Mio-> on each metric. For any metric with A, the raw value is instead reported. Rows for AFA (J), AFE (1), and AFMIA (L) present the absolute difference in the value of the corresponding method this metric the value of M the metric. These results show that RELOAD outperforms all the baselines on RA, AFA, AFE, AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other baselines other than FT, CF-k, and EU-k
27

---

Published as conference paper ICLR 2026
Method
AFA (L)
AFE (4)
APMIA
Cost
RSKL
FSKL (4)
GA
93.8110.75
18.77
1.
0.21 os
0.00
0.29 10.00
2.62
FT
96.00+0.12
16.46=2.45
0.89
0.19 os
0.27=0.00
0.03-20.01
2.11
SSD
1.01+0.02
74.17-2.04
4.19+0.59
0.15+a.22
0.01 0.01+0.00
14 1.290+1.24
11.81+1.24
SCRUB
93.7610.74
18.85=2.39
0.95
0.20ga.06
0.02g0.00
0.29.20.09
2.63.20.0m
CF-R
94.7510.411
18.01
0.94
0.201
0.21
0. 0.14110.3
2.47 20.07
EU-k
91.32+00.4
17.93=2.65
0.91
0.20
0.21
0.19
2.33 2.33=0.00
SalUn
99.0610.92
13.14_2.63
0.11
7.3912.00
0.16go.og
0.06
20.54
Fisher
97.7610.75
22.99
0.95
7.27
1.78
0.07
0.56 0.56
RELOAD (OURS)
99.56.2.11
0.30
0.04.2..22
0.01
0.12
0.15+0.08
1.23
Retrained (Baseline)
99. 99.98+a..1
74.89=2.05
1.06+0.13
0.63+a.20
Table 10% Random Forgetting on CIFAR-100 (ResNet-18). The bottom row presents the absolute value of Mi on cach metric. For any metric with A, the raw value instead reported. Rows for AFA (4), AFE (4), and AFMIA (4) present the absolute difference in the value of the corresponding method on this metric to the value of M on the metric. These results show that RELOAD outperforms all the basclines RA, AFA, AFE, AFMIA, and FSKL by large margins. RELOAD performs competitively on the RSKL metric, outperformed by FT and CF-k. RELOAD incurs a higher computational cost than most baselines, but cheaper than FT, CF-k, and EU-k.
B.4.2 RANDOM 30% FORGETTING
Method
RA (f)
AFA (L)
AFE
AFMIA (L)
AAUC
Cost
RSKL
FSKL (4)
30.20
77.46+26.25
8.86
0.02=0.02
0.01 #0.02
20.00
0.06+a.02
0.66
FT
99.6910.24
3.92+0.58
0.19+0.02
0.02=0.01
0.02+40.00
0. 0.28-:0.01
0.05
0.48
SSD
19.85
74.50:225.90 90
1.8220.58
0.01
0.01 20.02
0.01 10.00
8.30
7.83_2.70
SCRUB
82.59+1.99
12.72+1.61
0.31+0.04
0.00
0.00.go.00
0.07 +0.00
0.06+0.02
0.65-+0.04
CF-k
99 99.58
6.28:20.19
0.2724.01
0.05
0.05gg0.00
0.11 10.00
0.06g..022
0.55
99.5910.18
6.28.10.29
0.271.001
0.05
0.05.g0.01
0.22 01
0.07.10.02
0.56go.0:
SalUn
99.63
2.9740.50
0.37
0.02=0.02
0.02:0.02
0.20
0.06
0.55
Fisber
99 39.50+0.11
2.37+0.47
0,08++..1
0.02+0.00
0.02++0.0
1.79++0.0
0.07+4..02
0,56+0.04
RELOAD (OURS)
99.51
1.35
0.05
0.00
0.00go.00
0.30
0.12:a.02
0.53.0.07
Retrained (Baseline)
99.99.10.01
94.4010.71
0.23
0.50
0.50
Table 9: 30% Random Forgetting CIFAR-10(
t: the goal is to have as high value as possihle, A": the value in the table is difference hetween the result the unlearning method and retraining (hottom row) on the metric and the goal is to have a low difference, 4: the goal is to have as low of a value as possible. The bottom row presents the absolute value of M each metric. For any metric with A, the value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (1) present the absolute difference in the value show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other basclines other than FT, CF-k, and BU-k
corresponding method on this metric the value of M(e-; on metric. These results
28

---

Published as conference paper ICLR 2026
Method
RA (1)
AFA
AFE (4)
AFMIA
(A)
(4)
RSKL (L)
FSKL (4)
GA
18.97
73.93
0.4320.01
g.05
0.01
0.01
0.06
0.66
FT
20.37
4.4
0.27+0.02
0.02
0.02=0.00
0.27.20.01
0.05 01
0.48 04.04
SSD
22. 22.73+29..7
70.55+22.73
1.67+0.00
0.01
0.01 +0.00
0.01+0.00
8.30+3.11
7.83+22.70
SCRUB
14.29:25.22
77.1025.08
2.0220.57
0.01
0.01:20.00
0.08-20.00
0.06 02
0 0.65:20.04
CF-
99.4610.19
8.1610.
0.4010.02
0.05g0-01
0.05
0.15gg0-00
0.0610
0.55
EU-k
99 99.47=0.19
8.17
0 0.1022.02
0.05g0.01
0.03.20.00
0.30
0.07 02
0.56
SalUn
99.73 06
0.901..22
0.25110.0
0.01
0.01
0.18-20.00
0.06
0.33 04
Fisher
09.3740.00
3.6610.30
0.1340.01
0.0220.01
0.02
1.07.1.02
0.07 1.0000 oa
0.56.0.04
REI (OURS)
98.43++1.49
2.46+1.68
0.07+0.05
0.00
0.00
0.57++0.13
0. 0.1
0.53+0.07
Retrained (Baseline)
99. 00.93
94.40
0.23
0.50
0.50
Table 10: 30% Random Forgetting on MIEAR-I0(VGGI6-B)
t: the goal is to have as high of value as possible, A": the value in the table is the difference between the result the unlcarning method and retraining (bottom row) on the metric and the goal is to have a low difference, J: the goal is to have low of value possible. The bottom presents the absolute value of M(e-> cach metrie. For any metric with A, the value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (4) present the absolute difference in the value show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other baselines other than FT, CF-k, and EU-k.
corresponding method on this metric the value of M(g-> on the metrie. These results
Method
RA (f)
AFA
AFE (4)
AFMIA
AAUC
Cost (4)
RSKL (L)
FSKL (4)
GA FT
36.61 36.61
45.98
4.71
0.06
0.06
0.01
0.06 02
0.66 06
99.96 02
24.9410.00
1.0220.04
0.13
0.13
0.27.20.02
0.05 01
0.48
SSD
31.89+32.69
65.53++14.26
3.15+0.75
0.03
0.02+0.06
0.01
8.30+33.11
7.83+22.70
SCRUB
23. 23.96+2.22
48 18.86+2.22
1.97
0.01=0.01
0.01
0.07:20.00
0.06 02
0.65++0.04
CF-k
98 08.8510.00
21.3811.27
0.9210.04
0.1220001
0.11
0.10
0.06
0.55
98.30
20.1819.66
0.8910.04
0.11go.01
0.11g0.01
0.21g1o.0
0.07 02
0.56.0.00 04
SalUn
97 00.33
40.31#3.78
1.2040.04
0.10+0.01
0.10=0.01
0.20=#0.00
0.06
0.55
Fisber
97.76
1.54
0.0810.01
0.03
0.03
1.77 1.7720.00
0.07 2. oz
0.56 04
RELOAD (OURS)
99.56
1.47+1.06
0.0840.05
0.01
0.00
0.32.0.04
0.12
0.53
Retrained (Baseline)
19.98
94.4010.77
0.2312.08
0.50
0.50g0.00
Table 11: 30% Random Forgetting on CIFAR-100(ReSeet)
?: the goal is to have as high value as possible, A": the value in the table is the difference between the result the unlcarning method and retraining (bottom row) on the metric and the goal is to have a low difference, 4: the goal is to have as low of value as possible. The bottom row presents the absolute value of M( each metric. For metric with A, the value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (1) present the absolute corresponding method on this metric the value of M(g-> on metrie. These results
difference in the value show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other basclines other than FT, CF-k, and EU-k
29

---

Published as conference paper ICLR 2026
Method
RA (1)
AFA
AFE (4)
AFMIA
AAUC (J)
(4)
RSKL (L)
FSKL (4)
GA
10.75 87
62.76 35
2.1010.05
2.15
0.02
2.01
0.06
0.66
FT
98.30 68
15.86+1.84
1.40+0.06
0.06
0.06
0.28-20.01
0.05 01
0.48 04.04
SSD
31.72+32.14
62.23+12.59
2.43+0.19
0.02++0.04
0.02+00.6
0.01+0.00
8.30+3.11
7.83+22.70
SCRUB
1.60 20.86
65.
2.39:20.10
0.01.20.0
0.01:20.00
0.08-20.00
0.06 02
0. 0.65:20.04
CF-
97 97.61
29.8311..6
1.9510.04
0.14g0-01
0.14g0.01
0.15gg0-00
0.06
0.55
EU-k
97.71
29. 0,
1.95+0.04
0.14
0.14
0.30
0.07 0.
0.56
SalUn
98.8610.27
3.28
0.4210.04
2.00
1.00
0.18-20.00
0.06
0.33 04
Fisher
97.39
14.1940.8t
0.5610.02 02
0.07
2007
1.0620.02
0.07 1.000 oa
0.56.0.04
REI OAD (OURS)
88.95+9..2
8.94+5.71
0.18400.9
0.00
0.00
0.60+0.02
0.1
0.53+0.07
Retrained (Baseline)
99 99.8520.02
94.40
0.23
0.50
0.50
Table 12: 30% Random Forgetting on CIEAR- (GG-6--B))
t: the goal is to have as high of a value as possible, A+: the value in the table is the difference between the result the unlcarning method and retraining (bottom row) on the metric and the goal is to have a low difference, J: the goal is to have low of value possible. The bottom presents the absolute value of M(e-> cach metrie. For any metric with A, the raw value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (4) present the absolute difference in the value the corresponding method on this metric the value of M(g-> on the metrie. These results show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other baselines other than FT, CF-k, and EU-k.
Method
RA (f)
AFA (4)
AFE (4)
AFMIA (L)
AAUC (4)
Cost (4)
RSKL (4)
FSKL (J)
GA FT
36.70
59.35
6.22
0.02g0.03
0.02go.03
0.01g0.00
0.0611
0.66
100.00
4.73320.20
0.19
0.04
0.04
0,28
0.05 01
0.48.0.04
SSD
20.64
75.32+22.88
1.80
0.01
0.01 +0.03
0.01
8.30
7.83++2.70
SCRUB
97 07.234+..20
0.49:20.21
0.02:00.0
0.00
0.00=0.00
0.08
0.06
0.60
CF-k
100. 10.000
4.7922.22
0.19
0.05g0.01
0.0520.01
0.10go.00
0.06
0.55
99. 29.98
2.
0.18g0.0t
0.05
0.05
0.19 on
1:0 02
0.56.0.04
SalUn
99. 09.65+0.00
1.84+0.31
0.09
0.02=0.01
0.02=0.01
0.22=
0.06+
0.55++0.04
Fisber
99.62 19.622..14
20002
0.00
0.01
0.01
2.12go.03
0.07 1.
0.56go.04
RELOAD (OURS)
99.58 80
0.08
0.01
0.00
0.00
0.11g0.05
0.1240.01
0.53
Retrained (Baseline)
100.001..00 no
94.7240.19
0.25
0.50g0.00
0.50go.00
Table 13: 30% Random Forgetting on SeeN(Res
t: the goal is to have as high value as possible, A": the value in the table is the difference between the result the unlcarning method and retraining (bottom row) on the metric and the goal is to have a low difference, 4: the goal is to have as low of value as possible. The bottom row presents the absolute value of M( each metric. For metric with A, the value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (1) present the absolute difference in the value the corresponding method on this metric the value of M(g-> on the metrie. These results show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other basclines other than FT, CF-k, and EU-k
30

---

Published as conference paper ICLR 2026
Method
RA (1)
AFA (4)
AFE (4)
AFMIA (L)
AAUC (L)
Cost (4)
RSKL (4)
FSKL (U)
GA
g6 do
29.62226.16
0.25
0.05g0.03
0.01
0.01.0.00
0.0610
0.66
FT
100.00
4.84
0.23
0.03
0.03. 01
0.28
0.05
0.48.00.4
SSD
24.17+28.4p
71.83+25.00
1.80+0.00
0.01+0.02
0.01+0.02
0.01+0.00
8.30+3.11
7.83+2.70
SCRUB
24.26
70.72213.55
1.85.2..3
0.01:20.0
0.01.00.00
0.0820.00
0.06
0.65.20.04
CF-
99.60 99.60
4.84
0.23
0.04g0.01
0.04g0.01
0.12g0.00
0.0611
0.55
EU-k
99 60
1,16
0.23
0.01
0.01
0.25
0.07 02
0.50g2.04
SalUn
99.9110.04
0.81
0.04
0.00
0.00g0.00
0.19g0.00
0.0610
0.55.10.01
Fisher
99.53
0.04g0.03
0.00
0.00g0.00
0.00g0.00
1.43go.01
0.07
0.56g0.04
REI OAD (OURS)
99.37
0.10000.0
0.02++..1
0.00
0.00=0.00
0.15.0.01
0.1240.01
0.53+0..77
Retrained (Baseline)
100.00 Do
94.40.0.72
0.23
0.50
0.50g0.00
Table 14: 30% Random Forgetting on SVHN(YGG16-BN)
t: the goal is to have as high of value as possible, A"; the value in the table is the difference between the result the unlcarning method and retraining (bottom row) on the metric and the goal is to have a low difference, J: the goal is to have low of value possible. The bottom presents the absolute value of M(e-> cach metrie. For any metric with A, the value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (4) present the absolute difference in the value show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than other baselines other than FT, CF-k, and EU-k.
corresponding method on this metric the value of M(g-> on the metrie. These results
B.4.3 RANDOM 100 IN CLASS FORGETTING ADDITIONAL EXPERIMENTS
Methed
RA (f)
FA (A+)
(A+)
FMIA (A+)
Cost (L)
RSKL (L)
FSKL (4)
GA
99.57 99.57
2.37
0.1720.01
0.05
0.00
0.05g oo
0.52
FT
99.99 99.0910.00 10.00
2..3
0.17
0.04g0.01
0.27
0.001.0.0
0.43g0.0z
SSD
12.75-4.69 4.69
82.52+4.78
2.12+0.08
0.01+0.01
0.01
8.55+0.15
7.88+0..2
SCRUB
99.79
1.44:0.2
0.18
0.05
0.03
0.03
0.50 0.50=20.22
20
4.47:0.21
0.1820.01
0.05.g0-01
0.23 02
0.0322.00
0.50220.2
BU-k
09.63
2.4
0.18
0.05
0.23
0.05
0.47 20.02
SalUnn
99.90=0004
3.14+1.00
0.13=0003
0.04+0.02
0.17+0..0
0.03+0..0
0.50=0.02
Fisher
99.57 20.57
0.09.00.05
0.00
0.01
2.17
0.05 co
0.47
RELODAD (OURS)
99.68
0.25
0.01
0.00
0.12
0.06
0.21 2.2120.02
Retrained (Baseline)
99.9910000
96. 12....2
0.20
0.5020.00
Table 15: 100 In Class Random Forgetting on SVHN (ResNet-18)
t: the goal is to have high of a value possible, At: the value in the table is the difference between the result of the unlearming method and retraining (bottom row) the metric and the goal is to have low difference, J: the goal is have as low a value as possible. The bottom row presents the absolute value of M(e-> on cach metric, For any metric with A. the raw value is instead reported. Rows for AFA (L), AFE (L), and AFMIA (4) present the absolute difference in the value of the corresponding method on this metric to the value of M(a- on the metrie. These results show that RELOAD outperforms all the baselines AFA, AFE, AFMIA, and RSKL by large margins. RELOAD performs competitively RA and FSKL is outperformed by FT. RELOAD also incurs higher computational cost than the other baselines.
31

---

Published as conference paper ICLR 2026
Methed
RA
AFA
AFE (4)
AFMIA
Cost
RSKL (4)
FSKL (4)
GA
98.30
98.30.0.01 20.04
5.
0.21
0.04
0.00
0.07.0.00
2.63.0.00
FT
98.3840.15
8.1940.41
0.15+0.02
0.02+0.00
0.27 00
0.05
0.48
SSD
10.04-0.06
83. 15+0.87
2.07+0.02
0.00
0.01 +0.00
9.39+0.08
8,80 8.8006.06
98.33=20044
6.70.10.55
0.2220.01
0.05-20.00
0.02 no
0.07:20.00
0.63
CF-A
98.27
5.23
0.21
0.0520-01
0.23
0.0710
0.54220.02
28 20007
5.25-20.54
0.22=2.01
0.05:40.01
0.23 03.
0.07++0.00
0.52=0.03
SalUn
99.74.20.01
4.1110.45
0.2710.02
0.01:10-01
0.16
0.07.1
0.5420.02
Fisher
99.45
2.
0.0610.01
0.0220.00
1.78
0.07.00.00
0.5220.00
RAL (OURS)
97.00=11.0
3.46++0.8
0.08
0.01+
0.31 +0.09
0.110
0.52=6.00
Retrained (Baselinc)
92.81+0.52
0.24+0.01
0.50+0.00
Table 16: 100 Class Random Forgetting on CIFAR-10(ReNNet-151
ti the goal is to have as high a value as possible, A+; the value in the table is the difference between the result of the unlcarning method and retraining (bottom row) on the metric and the goal is to have low difference, J: the goal is to have low of value possible. The bottom presents the absolute value of M
each metric. For any metric with A, the value is instead reported. Rows for AFA (1), AFE (4), and AFMIA (4) present the absolute difference the value of the corresponding method on this metric to the value of M(a- on the metric. These results show RELOAD outperforms all the baselines on AFE. RELOAD performs competitively on RA, AFA, AFMIA, RSKI., and FSKL but is outperformed. FT which performs well, empirically makes little adjustment to the actual FA value. RELOAD also incurs higher computational cost than the other baselines.
Method RA (1)
AFA (L)
AFE (4)
AFMIA (L)
Cost (4)
RSKL (4)
FSKL (4)
GA FT
99.02+0.05
6.94+a.32
0.33+0.01
0.04+a.01
g.00
0.10
0.91+00.0
98.72ga.50
3.50
0.23
0.02gg.01
0.27g0.01
0.08 10.8
0.65
9.99 64
81.8820.00
2.1220.30 30
0.01
0.01
10.8810.78 79
10.25go.88
SCRUB
97.31
3.79.22.28
0.1440.08
0.0120.01
0.03-20.0
1.37:20.4b
1.75-2.45
CF-k
89.03
6,95.0.55
0.33
0.051g.01
0.37
0.10
0.7920.02
EU-À
99.02 65
6.96
0.33 01
0.0540.01
0.3720.08
0.10.20.00
0.7826.04
SalUIn
99.80+0.00
0.33
0.12
0.01
0.14g..00
0.100
0.70
99.32+a..5
3.81
0.10:4.01
0.02+0.00
1.07
0.10.20.00
0.78=0004
RELOAD (OURS)
98.5710.22
1.8821.62
0.1420.09
0.01
0.15g0.07
0.101:.:1
0.57
Retrained (Baseline)
99.5610.co
92.02g0.aa
0.3720.01
0.50
Table 17: 100 In Class Random Forgetting CIFAR-10 (VGG16-BN). The bottom presents the absolute value M(rr) each metric. For any metric with A, the raw value instcad reported. Rows for AFA (1), AFE (L), and AFMIA (1) present the absolute difference in the value the corresponding method on this metric to the value of M on the metric. These results show that RELOAD outperforms all baselines on AFA, AFE, AFMIA, FSKL indicating it behaves the closes to M(-- leading method by 0.46 for R.A and 0.02 for RSKL. RELOAD incurs higher computational cost than most baselines, but is cheaper than FT, CF-k, and EU-k. Other experimental settings are presented Appendix B.4.3
D forget. RELOAD performs competitively on RA and RSKL, falling behind of the
32

---

t: the goal is to have as high of value as possible, A": the value in the table is the difference between the result of the unlearning method and retraining (bottom row) the metric and the goal is to have low difference, J: the goal is to have low of value as possible. The bottom row presents the absolute value of M(er> on each metric. For any metric with A, the raw value is instead reported. Rows for AFA (J), AFE (4), and AFMIA (4) present the absolute difference in the value of the corresponding method on this metric to the value of M(g-> on the metric. These results Published as conference paper ICLR 2026
Method
RA (1)
FA (A*)
FE (A+)
EMIA (A*)
Cost (+)
RSKL. (L)
FSKL (4)
GA
98. 98.32 20-02
23.33 23.33
1.00
1.07
0.00
0.07
0.65
FT
0.
16. 16.84+1.08
0.8220.06
0.05
0.27
0.05-t0.01
0.48-20.04
SSD SCRUB
10.01
68.67
5.75
0.38.
0.00+0.00
9.33 os
8.72
98.35_0.00
27.55
1.0220.06
0.07
0.02
0.07
0.65 0.00
CF-A
18.1
21.84+0.85
0.99+0.05
0.07++0.08
0.21
0.07+00.00
0.54-20.44
EU-k
98. 98.24=20.0m
21. 21.95+00.7
0.99
0.07-+0.08
0.21 0.21+t0.01
0.07++0.00
0. 0.50=20.44
99.57 20.02
12.08-13.13
0.48g0.or
0.02.20.02
0.14 no
0.07.20.00
0.54g0.05
Fisher
97 97.50
10.72:11.9
0.1910.04
0.0311.04
1.81
0.07
0.0022004
RELOAD (OURS)
99.17 +0.09
3.44
0.20+0.a5
0.02
0.26+0.11
0.12#0.01
0.53=0.08
Retrained (Baseline)
95.50-00.24
70.05+1.99
1.13+0.07
0.83++0.20
Table 18: 100 In Class Random Forgetting on CIFAR CIFAR-IO0(ReeNer-1BB))
show that RELOAD outperforms all the baselines on RA, AFA, AFE, and AFMIA, by large margins. RELOAD performs competitively on RSKL and FSKL but is outperformed by FT. RELOAD also incurs higher computational cost than the other baselines.
Method
RA (1)
FA (A*)
FE (A+)
FMIA (A+)
Cost (L)
RSKL (L)
FSKL (1)
GA
20.31
28.55
1.70 04
0.03
0.00110.00
0.07.
0.65
FT
98. 14-0225
11.44+1.77
1.07
0.01 20.01
0.28
0.06::n.0
0.47 0..772..00
SSD
10.00 20.00
63.8642.112
2.70.0.15
0.45-g0.04
0.00
9.3620.05
8, 8.75
SCRUB
98.33_0.02
30.5911.25
1.7620.05
0.0420001
0.02
0.0710.00
0.63
CF-k
0.
26 16
1.7520.07
0.01go..0
0.34 07
0.07.0..00
0,54
BU-k
98.22
25.3711.39
1.682o.o8
0.03.go.o2
0.33
0.07.1
0.55
SalLin
99.40_0.04
7.56
0.31
0.00g0.00
0.13
0.07
0.54g0.00
Fisher
00.00
19.5544.55
0.67=0.05
0.03
1.05 04
0.07
0.55-0.08
RELOAD (OURS)
99.47 :0.04
1.84
0.14g0.04
0.03
0.29
0.12+0.01
0.51+0002
Retrained (Baseline)
93.85.21.0
65.2642216
1.95
0.93
Table 19: 100 In Class Random Forgetting on CIFAR-100(YGG16-BN)
t: the goal to have high of a value possible, At: the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and goal is to have low difference, the goal have as low of a value as possible. The bottom row presents the absolute value of M on each metric. For any metric with A, the raw value is instead reported. Rows for AFA (4), AFE (4), and AFMTA (4) present the ahsolute difference in the value of the corresponding method on this metric to the value of M(a-) RELOAD outperforms all the baselines RA, AFA, and AFE by large margins. RELOAD performs competitively on AFMIA, RSKL and FSKL but is outperformed by FT. RELOAD also incurs a higher computational cost than the other baselines.
the metrie. These results show that
33

---

Published as conference paper ICLR 2026
Method
RA (1)
FA (A+)
FE (A")
FMIA (A*)
Cost (4)
RSKL (4)
FSKL. (4)
GA
99.57
4.4610
0.22
0.03
0.0040.00
0,05
0.51
99.9940.001 000001
4.47+0.22
0.22:0.01
0.03
0.27:2000
0.00
0.43 02
SSD
14.55
84.1911.55
2.03
0.00
0.01g0.00
8.5110003
7.84
SCRUB
99.79
9.
0.36.10.34
0.03
0.02g0.00
0.03....0 00
0.50 08
CF-R
99.76+0.00
4.53.+0.25
0.23-+0.01
0.04+0.01
0.24+0.02
0.03-+0.00
0.50 02
EU-k
99. 0..02
4.54.+..2
0.23-+0.a1
0.01
0.24::0.2
0.00-20.00
0.47+0.00
SalUn
99.94.0.00
3.04
0.16:20.01
0.03.20.01
0.1420.01
0.03.10.00
0.50 02
Fisher
99.48
0.091
0.00
0.00
1.4210.14
0.05g0.00
0.47
RELOAD (OURS)
99. 00.67
0.93
0.05
0.01
0.14+0.09
0.06+#002
0.21:0.02
Retrained (Baseline)
99. 09.999+4..00
96.09+0.19
0.20++0.01
0.50+0.00
Table 20: 100 In Class Random Forgetting on SVHN (VGGI6-BN)
t: the goal is to have as high of a value possible, A+: the value in the table is the difference between the result of the unlearning method and retraining (bottom row) the metric and the goal is have low difference. the goal have low with A, the raw value is instead reported. Rows for AFA (L), AFE (J), and AFMIA (4) present the absolute difference in the value of the corresponding method on this metric to the value of M(a~) on the metric. These results show that
value possible. The bottom row presents the absolute value of M(s-) each metric. For any metric
RELOAD outperforms all the baselines on AFA, AFE, and FSKL, by large margins. RELOAD performs competitively RA, AFMIA, and RSKL but is outperformed FT. RELOAD also incurs higher computational cost than the other baselines.
B.5 LANGUAGE MODEL ENTITY UNLEARNING RESULTS
Unlearning for language models (LMs). When D corpus of texts, we express forgetting via set of prompts D prompts that target concepts or entities in D forget, and possibly small repair set Deepoir S Dretain.
The results presented below are taken from prior work (Liu et al., 2024a) with results for RELOAD appended the bottom due to computational constraints. In these result tables, the gold-standard retrained model denoted 'Retain'.
Due computational constraints and the lack of open-source retrained models for Phi-1.5 in the 1 % and 5% forgetting case, our results for Phi-1 1.5 are limited to the 10% forgetting case.
34

---

Published as conference paper ICLR 2026
Split
Method
Change in Model Utility from Original
Forget Quality
Original
+0.0000
0,0030
Retain
-0.0131
1.0000
Grad Ascent
-0.0233
0,0068
Grad Diff
-0.0198
0.0143
%
KI. Min
-0.0221
0.0068
Pref Opt
-0,0021
0,0971
Prompt
-0,0628
0.0068
NPO
-0.1725
0.7659
NPO-KL
-0.1703
0,4046
NPO-RT
-0.1361
0,5786
ECO (Rand Noise)
+0.0000
0,9188
ECO (Zero-Out)
+0.0000
0.9900
ECO (Sign-Flip)
+0.0000
0.0002
RELOAD (OURS)
+0.0748
0,4046
Original
+0.0000
0.0000
Retain
-0,0229
1 1.0000
Grad Ascent
-0.6257
0.0118
Grad Diff
-0.3013
0.0000
5%
KL Min
-0.6257
0.0163
Pref Opt
0.1142
0.0000
Prompt
0..003
0,0000
NPO
-0,4512
0.7934
NPO-KL
-0,2203
0.7934
NPO-RT
-0.0838
0.6284
ECO (Rand Noise)
+0.0000
0.9647
ECO (Zero-Out)
-0.0009
0.9647
ECO (Sign-Flip)
+0.0000
0.0000
RELOAD (OURS)
-0.2870
0.5453
Original
0.0000
0.0000
Retain
-0.0160
1.0000
Grad Ascent
-0.6257
0.0000
Grad Diff
-0.0434
0.0000
10%
KL Min
-0,6257
0,1810
Pref Opt
-0.0862
0.0000
Prompt
-0.1380
0.0000
NPO
-0.4556
0.0126
NPO-KL
-0,2634
0,0158
NPO-RT
-0. 1260
0.0783
ECO (Rand Noise)
-0.0028
0.5812 12
ECO (Zero-Out)
-0.0014
0.9674
ECO (Sign-Flip)
-0,0022
0,0000
RELOAD (OURS)
-0.3384
0.7000
Table 21: Change in Model Utility and Forget Quality of different unlearning methods on unlearning entities from the TOFU dataset
35

---

Published as conference paper ICLR 2026
Split
Method
Change in Model Utility from Original
Forget Quality
Original
0.0000
0,0000
Retain
+0.0053
1,0000
Grad Ascent
-0.5518
0,2107
Grad Diff
-0. 0999
0,0000
10%
KI. Min
-0.5518
0,4158
Pref Opt
-0.0379
0.0000
Prompt
-0,0363
0,0000
NPO
-0.3669
0,0013
NPO-KL
-0.2520
0.0049
NPO-RT
-0.1144
0.7000
ECO (Rand Noise)
-0.0003
0,8635
ECO (Zero-Out)
-0.0031
0.9844
ECO (Sign-Flip)
-0.0001
0.0446
RELOAD (OURS)
-0.3384
0,4680
Table 22: Change Model Utility and Forget Quality different unlearning methods on unlearning entities from the TOFU dataset on Phi-1.5
B.6 CORRECTIVE UNLEARNING RESULTS
Baselines. The corrective unlearning setting admits different baselines than the unlearning setting based on prior work (Goel et al., 2024). RewoD represents a baseline model trained directly on Dretain-
Evaluation. We evaluate corrective unlearning following Goel et al. (2024). The corrected accuracy Accaur measures the performance the unlearned model on the adversely affected data, Dm The retain accuracy
AcCietain measures unlearned model performance on a held-out validation sample of Dretain, D Petaiin Cost measures the runtime of the algorithm comparison retraining (Table
Reload efficiently corrects trained models. We evaluate RELOAD 's ability
unlearn adverse effects of manipulations following the baselines outlined prior work (Goel et al., 2024). We present results for RELOAD and unlearn- ing baselines on the two conventional corrective unlearning tasks, Poisoning and Interclass Confusion (IC), as well as the corrective unlearning (with re- placement) settings introduced in Appendix ??. The results of this experiment over different settings presented in Table 24 and Figures 9, 10, 11, and 12 in Appendix ?? for consistency with prior work. RELOAD outperforms baselines Acc.ur at low percentages of data identification (Figures 8a, 8b) while observing competitive computational efficiency (Table 23), even at only 10% data identification (7 0.1). Across 10 values of from the corrective setting (Goel et al., 2024), RELOAD consistently outperforms all other baselines in most experiments. Although BadT (Chundawat et al., 2022) outperforms RELOAD CIFAR100 Poisoning experiments, it bears much greater computational cost (Table 23).
Image
Image
Figure Data poisoning serts the patterns (right) in all selected images in D.
36

---

Published as conference paper ICLR 2026
100
SSD
Rekond
Scrub
SSD
CP
60
Senih
RoweD
CF
Ankerr
DaaTT
RoweDD
Reboad
B
20
0.2
0.1
0.6
0.8
1.0
0.2
0.3 Ieeiliiedd Fraction 7
0.6
0.8
100
Inlentified Praction
(a) [D... 100
(b) [D..] 2500
Figure 7: Corrective Accuracy (AcCem)) after applying different unlearning methods. This measures the performance of the unlearned model the domain representing the adversely affected data, Dn measures the proportion of the adversely affected data which identified and collected within D We that small y, RELOAD achieves consistently higher Acc.or than existing baselines and performs across ry values.
SSD
Roload
Seraly
SSD
CP
60
Serth
RewnD BaiTT
CP
Ancorr
Rewal
Relood
Bod
20
022
0.4
0.6
6.8
1.0
0.2
0,4 Idmiiiie Fraction 7
0.R
1.0
ldcutified Freezion
|Dmll 100
(h) [Dml 2500
Figure 8: Corrective Accuracy (Accem) after applying different unlearning methods. This measures the performance of the unlearned model on the domain representing the adversely affected data, Dn. y measures the proportion of the adversely affected data which identified and collected within forget- We note that small 7. RELOAD achieves consistently higher Acc.or than existing baselines performs across 7 values.
SSD
SSD
SSD
Scrub
Scrth
Sentb
ReweD
Sevol
Resad
Rei'T
Buff!
022
0.6
0.8
0.2
34
0.8
100
0.2
B1
0.8
10
Rustion
Puotion
Frotion
(a) (Dm 500
(b) DDal 2500
(c) Dmal 5000
Figure 9: CIFAR10 Interclass Confusion
37

---

Published as conference paper ICLR 2026
350
100
SSD
[0]
SSD
HewoL)
Hewoll
RewoD
Rehood
Rickod
Bekaa.
Mraaifed Fratan
Meniifiel Framacn y
Fonedial Fraction
(a) [Dm 100
(b) Pm 500
(c) [ 1000
Figure 10: CIFARIO Poisoning
100
ReweD)
Rewel
BrooD
Scrada
Scndl:
Senbb
F01
RecT
ReilT
Doff
Relood
Telvol
Bekeal
0.2
CA4
0.5
0.2
3.4
0.6
0.8
0G Ranedial
Iderified
(a) [Dml
100
(b) Dmn 250
(c) 1mml 500
Figure 11: CIFAR100 100 Interclass Confusion
100
S80
300
S80
100
SSD
BowoD
RanolD
BranD
Scndh
Senjo
Scrdh
CP
CI
Balg
022
Meatieed Frodoo
Meciillied Hrationrry
looeiited Etatioony
(a) [
100
(b) Dmm] 500
(c) Dmml 1000
Figure 12: CIFARIO0 Poisoning
Haff
Bofl
CV
Serth
Edoed
Rowa
Iickad
lirk ut
Meatiieed Frackks
Routiiie Baalicn
Momitie Proticn
(a) ( m
(b) DD.. 2500
(c) P... 5000
Figure 13: CIFARIO Interclass Confusion (with replacement)
38

---

Published as conference paper ICLR 2026
Bamill
BowDD
CVV
UP
00
ISD
Ricbad
Fdlied
Rchaad
0CB
000
(a) [Dm] = 100
(b) Din] 500
(c) [Dml 1000
Figure 14: CIFARIO Poisoning (with replacement)
100
RenolD
BowDD
BoxaD
Barff
Bedead Buffil
Ball
Semis
Scrhh
Antur
SBD
SSD
Mectiied Frackes
Rounitod Eactico
Dkantited Protion
[Dm
100
(b) Dmn 250
(c) [Dml 500
Figure 15: CIFARI00 Interclass Confusion (with replacement)
ISD
CV
Ball
Senvo
Bofl Scrub
Serih
Rilial
RED
022
RC
10
Eaaiioor
Pmotionry
(a) [ 100
(b) D... 500
(c) DDnll 1000
Figure 16: CIFAR 100 Poisoning (with replacement)
39

---

Published as conference paper ICLR 2026
(L)
CIFARIO
CIFARIO0
Cost (t)
CIFARIO
CIEARIO0
Cost (1)
CIFARIO
CTFAR100
Poisoning
Poisoning
Poisoning
RadT
0.44
0.68
RadT
0.464001
0.68+aum
BadT
2.46
0.68+4.00
0.26
0.25 1000
0.27
gamm
CF
0.27 1000
g.25
SSD
0.04.2000
0.06
SSD
0.04:10m
0.06
SSD
0.04 0.04-1one
1.06
Scrub
0,26-2ac0
0.31 2000
Scrub
0.27 2:00
0.31
Scrub
0.27 2one
+:000
RewoD
1.00+a.c
1,00+000
RewoD
1.00
1.00.4.00
RewolD
1.00
1.00+-0,00
RELOAD
0.37
n024
RELOAD
0.29 2o0t
0.25
RELOAD
g.33
0.25
Interclass Confusion (IC)
Interclass Confusion (IC)
Interclass Confusion (1C)
BadT
0.42
0.68
BadT
0.46
0.68
BadT
g.43
0.68
CF
0,23
0,26
CF
0.27.10m
0.25 gmm
CF
0.27 2.00
0.25
SSD
0.04gg00
0.06
SSD
0.042am
0.06
SSD
0.04-2one
0.06:20.0
Scrub
0,24++000
0.31
Scrub
0.27++0m
0.31
Scrub
0.27
0.31 +0.00
RewoD
1,00=
1.00
RowoD
1.00
1.00
ReweD
1.00=00e
1.00
RELOAD
0.28
0,14
RELOAD
0.40
RELOAD
0.1 0.14
0.14
Num. Corrupted Samples
Num. Corrupted Samples
Nom. Corrupted Samples
CIFARIO
Poison
100
CIFARIO CIFAR 100
Poison
500
CIFARIO CIFAR 100
Poison
1000
CIFARIOD
Poison
100
Poison
500
Prison
1000
CIFARIO
IC
500
CIFARIO
IC
2500
CIFARIO
IC
5000
CIFARIOD
100
CIFARI00
250
CIFARIOO
IC
500
Table 23: Cost (1) values across different sizes D Forget (Corrective Unlearning). Results are reported mean over values of 7.
Accyetii (t)
CIFARIO
CIFARIO0
Accymohn (1)
CIFARIO
CIFARIOD
Accyasi (1)
CIFARIO
CIFARI00
Polsoning
Poisoning
Polsoning
None
91.35
74.05
None
90.97
74.20
None
90.84
74.34
BadT
0.13
0.13
BadT
-0.05
-0.32.p011
BadT
0.01
-0.24
0.09--a12
0,54+011
0.35
0.32-+024
CF
0.27+auu
0.39+0.19
SSD
.3.05
2.088
SSD
-14.03
0.00
SSD
-0.66
-0.61 -0.61
Scrub
g.25
g.43
0.20
0.27
0.00
RewoD
0,80
1.14
RowoD
1,25 1:25::000
0.71 1:00
RewoD
0.92
1.18 1.18
-7.83++0.0
-13.20+009
-8.14-0022
-13.22+054
-7.55+0244
-13.63 +3si
Interclass Confusson (IC)
Interclass Confusion (IC)
Interclass Confusion (IC)
93.01
73.82
None
None
92.81
73.81
BadT
0.39=a.e
0.22
BadT
0.81
0.05
BadT
0.52+a.1
-0.02
CF
0.16
0.63
CF
0.59
0.46
CF
0.491031
0.53
SSD
-1.451444
0,14-1nc0
SSD
0.73
0.00 0.00
SSD
0.72
0.00
Scrub
0.19-=019
0,17
Scrub
0.63
-0.06
Sembb
-1.10
-0.05
0.82+#a00
+029+000
1.14
0.99 +000
0.95
1.13+am
RELOAD
0.200.om3
4.45
RELOAD
1.
5..2
RELOAD
-1.14
4.93
Num. Corrupted Samples
Num. Corrupted Samples
Num. Corrupted Samples
CIFARIO
Paison
100
CIFARIO
Poison
500
CIFARID
Poison
1000
CIFARIOD
Poison
100
CIFARI00
Poison
CIFARI00
Poison
1000
CIFARIO CIFARIOD
IC
500
CIFARIO
IC
2500
CIFARIO
IC
5000
Table 24: Accresia (f) values across different sizes Dj forget (Corrective Unlearning). Results are reported as mean over values of
40

---

Published as conference paper ICLR 2026
(L)
CIFARIO
CIEARIO0
Cost (1)
CIEARIO
CTEARIO
Cost (1)
CIEAR1O
CIFARI00
Poisoning
Poisoning
Poisoning
RadT
0.63 +0.01
0.69 =0.00
RadT
0.64 +0.01
0.70 +0.00
BadT
0.64 =0.01
0.68 +0000
0.26 10.01
0.25 20.00
0.26 10.01
0.25 10.
CF
0.25 =0.01
0.25 4:0
SSD
0,06 20.01
0.06 20.00
SSD
0.06 10.01
0.06 10.00
SSD
0.06 =0.00
0.06 4:0
Scrub
0.27 t0.01
20.00
Scrub
0.28 10.02
0,31 40.00
Scrub
=0.01
0.30 10000
RewoD
1,00
1.00
Rewol
1.00
1,00
RewolD
1.00
1.00
RELOAD
20.08
20.00
20.02
+0.00
=0.02
=:000
Interclass Confusion (IC)
Interclass Confusion (IC)
Interclas Confusion (IC)
BadT
0.62 +0.01 01
0.68 =0.01
BadT
0.63 #:000
0.67 +0.0
BadT
0.64 =0.02 02
0.68 4:0 00.00
CF
0,25 20.01
0.25 10.00
CF
0.27 10.01
0.24 10.00
CF
0.27 20.01
0.25 1:0 00
SSD
0.06 10.01
0,06 20.00
SSD
0.06 1:000
0.06 10.00
SSD
20.01
0.06 1:0.0
Scrub
0,27
0.30
Scrub
0.27 +0.01
0.30
Scrub
0.27
0.31
RewoD
1,00 +0.04
=0.04
RowoD
1.00 +0.05
1,00 +0000
ReweD
1.00 =0.04
1.00 +0.00
RELOAD
0.35
0.20
RELOAD
0.46
0,20
RELOAD
0.24
0.20
Num. Corrupted Samples
Num. Corrupted Samples
Num. Corrupted Samples
CIFARIO CIFARIOD
Poison Poison
100
CIFARIO
Poison
500
CIFARIO
Poison
1000
100
CIFAR 100
Poison
500
CIFAR I00
Poison
1000
CIFARIO
IC
500
CIFARIO
IC
2500
CIFARIO
IC
5000
CIFARIOD
IC
100
CIFARI00
IC
250
CIFARIOO
IC
500
Table 25: Cost (4) values across different sizes of 'D/ forget (Corrective Unlearning replacement). Results are reported
as godne over 10 values of 7
Accyetii (t)
CIFARIO
CIEAR100
Accymoin (1)
CIFAR10
CIFARI00
Accycsia (1)
CIFARIO
CIFARI00
Polsoning
Polsoning
Poisoning
None
91.35
74.05
None
90.97
74.20
None
90.84
74.34
BadT
-0.02 10.00
-0.13 20.15
BadT
-0.35 10.14
-0.72 20.42
BadT
-0.19
-1.37
+0.09
0.37 +0.07
+0.11
+0.11
CF
+0.
0.38 +0.11
SSD
16.00
0.00
SSD
40.73
0.00
SSD
-0.00
-73.34 4:0
Scrub
10.01
-0.01 10.00
10
10.17
1: 12
10.16
RewoD
0.72 20.10
1.31 20.11
RowoD
1.31 10.00
0.94
10.20
ReweD
1.11
0.96
+0.42
+0.01
+0.35
40.01
=0.29
+0..15
Interclass Confusion (IC)
Interclass Confusion (IC)
Confusion (IC)
None
93.01
73.82
None
92.22
None
92.81
73.81
BadT
0.39 =0.16
0.03 +0.11
BadT
0.83 #0.12
.0.23 =0.04
BadT
0.56 10.18
-0.15 40.12
CF
.0277 10. 22
0.48 1..
CF
-0.18 10.18
0.14 1: 26
CF
-0.07 20.08
0.15 1.0
SSD
-44.46 121.45
-72.80 10.00
SSD
-21.52 1:2..00
0.00 00
SSD
-1.23 21.63
-72.79 1:0 oo
Scrub
0.38 =0.01
0.12
Scrub
0.99 20.25
-0.05
Scmbb
-17.88
-0.07
=0.10
1.30 +0.10
+0.24
+0.11
+0.00
+0.18
RELOAD
5.84
30.63 116.14
RELOAD
0.01
22.25
RELOAD
-3.02
26.35
Num. Corrupted Samples
Num. Corrupted Samples
Num. Corrupted Samples
CIFARIO
Poison Poison IC
100
CIFARIO
Poison
500
CIFARID
Poison
1000
CIFARIOD
100
CIFARI00
Poison IC
500
CIFARI00
Poison
1000
CIFARIO
500
CIFARIO
2500
CIFARIO
IC
5000
CIFARIOD
CIFARIO0 100
Table 26: AcCussin (1) values across different sizes DD forget (Corrective Unlearning with replacement). Results are reported as m over 10 values
41

---

Published as conference paper ICLR 2026
B.7 HYPERPARAMETER SELECTION
RELOAD admits 4 hyperparameters. Additional hyperparameters may be introduced depending on the optimisation procedure used by the practitioner for RELOAD (eg. weight decay).
1. Alpha (a): The quantile of weights to reinitialise
2. Ascent Learning Rate: The step size for the ascent stage of RELOAD
3. Finetuning Learning Rate: The step size for the finetuning stage of RELOAD
4. Weight Reset Method: The scheme to use for reinitialising weights
B.7.1 WEIGHT RESET/REINITTALISATTON METHODS
First, we detail the different weight reinitialisation methods we explore options for the resetting step of RELOAD. This setting hyperparameter of RELOAD.
Mean. The selected parameters are replaced with the mean value of the tensor they are part of.
Zero. The selected parameters are replaced with the value
Normal. The selected parameters are replaced with random number drawn from (N 1). Uniform. The selected parameters are replaced with random number drawn from (((-1, 1). Xavier Uniform. The selected parameters are replaced with values obtained through Xavier Uniform initialisation (Glorot Bengio. 2010).
Xavier Normal. The selected parameters are replaced with values obtained through Xavier Normal initialisa- tion (Glorot& Bengio, 2010).
Kaiming Uniform. The selected parameters are replaced with values obtained through Kaiming Uniform initialisation (He et al., 2015).
Kaiming Normal. The selected parameters are replaced with values obtained through Kaiming Normal initialisation (He et al., 2015).
B.7.2 HYPERPARAMETER FOR CLASSICAL UNLEARNING
We train ResNet-18 and VGGI6-BN models on CIFAR-10 (Krizhevsky, 2012), CIFAR-100 (Krizhevsky et al.). and SVHN (Netzer et al., 2011) for image classification for 182 epochs. We apply the cross-entropy loss function and learning rate of 0.1 with a batch size of 256. We conducted these experiments over 10 random seeds to obtain average results and standard deviation measurements. The results in our tables are reported in the format Pio where a is the average value and is the standard deviation, across the 10 seeds.
The 10 seeds we selected for unlearning experiments were seeds 2,, 4, 5, 6, 7, 9, 10).
Hyperparameters for RELOAD were chosen through bayesian hyperparameter sweep. The chosen hyperpa- rameters for the unlearning tasks are presented in Table 27.
We empirically find that the cumulative distribution function of the knowledge-value for forgetting 10% of data from a ResNet-18 18 model trained on SVHN forms a sigmoid-lik curve around 10-1, This further evidences the existence of clear differences in the knowledge-values for different parameters. Experimentally. we select the thresholding hyperparameter using hyperparameter sweep. We have included in ablation (Appendix ??), a study with varying learning rates (r)) and thresholds (a).
42

---

Published as a conference paper ICLR 2026
Experiment
Alpha (o))
Ascent Learning Rate
Finetuning Learning Rate
Weight Reset Method
SVHN + R8sNet-18
0.1
0.243
0.098
Uniform
SVHN VGGI6-BN
0.1
0.496
0,496
Xavier Uniform
CIFAR-IO R8sNet-18
0.1
0.44
0.33
Xavier Uniform
CIFAR-10 VGGI6-BN
0.1
0.167
0.39
Kaiming Uniform
CIFAR-I00 + ResNet-18 18
0.1
0.18
0.33
Xavier Normal
CIFAR-100 + VGGI6-BN
0.1
0.325
0.164
Kaiming Normal
Table 27: RELOAD Hyperparameter Scttings for Unlearning
Baseline Implementations. Implementations for baselines were taken from the reference implementations for SCRUB, SSD, EU-ki and CF-k. Implementations for FT and GA were taken from the repository for SalUn.
Codebase Structure. Our codebase is built on the publicly-released repository for SalUn (Fan et al., 2023).
B.7.3 HYPERPARAMETER FOR LANGUAGE MODEL ENTITY UNLEARNING
For our base models we employ open-source model weights fine-tuned on the TOFU dataset publicly available on HuggingFace for Llama-2-7b-Chal (Touvron et al., 2023) and Phi-1.5 (Li et al., 2023). We use open-source fine-tuned models available on Hugging Face (locuslab, 2025; Unlearning, 2025a;b;c) our gold-standard retrained models.
Hyperparameters for RELOAD were chosen through a bayesian hyperparameter sweep. The chosen hyper- parameters for the unlearning tasks are presented in Table 28. All experiments were conducted using the AdamW optimizer from PyTorcc.
As discussed, RELOAD for LMs is parameter -fffeccent, and operates on a subset of the model parameters in a language model. As such, the entire RELOAD process is performed over a subset of the layers the language model. We selected this layer a hyperparameter through sweep, and report it as Target Layers in the below table. This structure enabled effective unlearning, and also increased the efficiency of the algorithm as gradients were only computed for certain layers. This allowed RELOAD to operate large language models on less powerful hardware setups and less time.
Experimone
Alpha (v)
leaming Finetming Laminng
Sumple
Beget Layes
Repair Epechs
Limma
Foegetil
0/119
152
Llama 2-Th Clat a Foegetts
0.017
3.049
0.0002
Unifomn
193
sulf satt_n Kupeo weiges
Ulmma-o--h Dama----h--hhm Th-Chrt Peeget00
01305
0.02
Xastze CesterUnten Unikemn
NI
self acenig prod ligh
78
1.5 i Pogel1
0.396
0.094
0.8443
Uzifonn
30
solf_sttnn Pupeol neigus
40
Table 28: Hyperparameter Settings for LM Entity Unlearning
Baseline Implementation and Results. Baseline implementations and results were obtained and reused from the repository and paper of Liu et al. (2024a).
Codebase Structure. For LM Entity Unlearning we reuse the publicly-released repository for Large Language Model Unlearning via Embedding-Corrupp Prompts (Liu et al., 2024a) to which we add our implementation of RELOAD for LMs.
B.7.4 HYPERPARAMETERS FOR CORRECTIVE UNLEARNING
We train ResNet-9 models CIFAR-10 (Krizhevsky, 2012) for 4000 pretraining iterations. We train ResNet- 28x10 models on CIFAR-100 (Krizhevsky et al.) for 6000 pretraining iterations. We apply the cross-entropy loss function, batch-size of 512, and a learning rate of 0.025. The results in tables are reported in
43

---

Published as a conference paper ICLR 2026
manner consistent with prior work (Goel et al., 2024) across selections of where y is the proportion of the forget set identified.
Hyperparameters for RELOAD are chosen through Bayesian Hyperparameter sweep. The chosen hyperpa- rameters are presented in Table 29. Hyperparameters for baselines are chosen through grid search defined the reference implementation. For all experiments we employ the SGD (Stochastic Gradient Descent)
optimizer with momentum 0.9 and weight decay 0.0005.
Experiment
Alpha (rx)
Learning
Finetuning Learning Rate
Weight Reset Method
CIFARIO ResNet-9 Poisoning
0,3984
0.01
0.00381
Xavier Normal
CIFAR10 + ResNet-9 9 Interclass Confusion
0.1978
0.01
0.00957
Mean
CIFAR100 ResNet-28x110 Poisoning
0.2401
0.00483
Xavier Normal
CIFARI00 ResNet 2 Interclass Confusion
0.0924
0.01
0.00237
Zero
Table 29; RELOAD Hyperparameter Settings for corrective unlcarning (with and without replacement)
Baseline Implementation. Baseline implementations are taken from the reference implementations provided in the repository for Corrective Machine Unlearning.
Codebase Structure. For corrective unlearning we reuse the publicly-released repository for Corrective Machine Unlearning (Goel et al., 2024) to which we add our implementation of RELOAD. To this repository, we also add our implementation corrective unlearning (with replacement) experiments.
B.8 HARDWARE USAGE
Hardware for Classical Unlearning All experiments were 4 CPU cores, 20 GB of RAM, and 1 NVIDIA T4 GPU.
Hardware for Language Model Entity Unlearning All experiments were run on 30 CPU cores, 60GB of RAM, and 1 NVIDIA A40 GPU. Some experiments were also conducted using 1 NVIDIA RTX6000 GPU to highlight the lightweight nature of RELOAD for LMs.
Hardware for Corrective Unlearning All experiments were 4 CPU cores, 60 GB of RAM, and NVIDIA RTX6000 GPU.
C FURTHER ABLATIONS
C.I ABLATION: CRITICAL COMPONENTS OF RELOAD
designing the knowledge values for identifying knowledgeable parameters, considered several other approaches in addition to the final formula (Eq. 1). This includes normalising gradients and utilising cosine similarity for the computation of knowledge values.
Ascent Steps. In designing the ascent step, we considered the possibility of needing multiple steps to appropriately scrub the global information from the model parameters. Theoretically, this notion violates the partially-blind nature of the unlearning setup, and was thus undesirable. Empirically, we noted that using multiple ascent steps does not improve forgetting and can lead to further performance degradation on D, Dratain requiring more retraining to get to final unlearned model. As it is the only partially-blina variant, we include a study RELOAD when the ascent step is not applied below.
44

---

Published as conference paper ICLR 2026
C.2 ABLATION: EMPIRICAL EVIDENCE
Below we present short ablation study on 3 different variations of components of the RELOAD algorithm.
1. ReloaddWithutAAsceel This the same as the standard RELOAD algorithm without the ascent step
2. Reload WithoorWWioooaiiiitioion This variant employs gradient normalisation before the calculation of knowledge values to increase directional information and reduce scaling issues
3. ReloadWithCosineK V: This variant uses cosine similarity between gradients to compute knowledge values instead of gradient magnitudes
We demonstrate these variants against the baselines RELOAD algorithm corrective unlearning tasks. Select results for corrective unlearning are shown in Figure 17.
24
ReloadWiihhuut.Aee
80
Relow//iithoutteecuu
ReloudWidhNorrrulssttiin
Relovl
22
Reload
Reload SedoodWitte Withooommmmiittiioonn
Reloed CoiConnck
G0
ReloadW WithCossnncV
220
Aptour
40
18
20
14
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Ielentffied Fraction
Identified Fraction
(a) CIFARIO Example 1
(b) CIFARIO Example 2
80
ReloudWichouttssce
16
ReleadWithont
Reload
Reload
70
ReloaddWithNootaalllae Normaastiiioiiionnn
Reload WithNormaaiiittoonnn
ReloedWithhoshhefe
ReloadWithhhssinee
60
AnCort
AcCare
12
50
40
10
30
20
0.6
0.6 Identified Fraction
Identified Fraction
(c) CIFAR100 Example
(d) CIFAR100 Example
Figure 17: Corrective Accuracy (Acc.aor) across variants of RELOAD.
the case of corrective unlearning, we note that the variants of RELOAD perform comparatively to the base algorithm but all note significant weaknesses in comparison. ReloadWithCosieet produces unlearned models with higher utility (higher ACCretains Table 18) but significantly lower corrective accuracy (Acccorrs Fig. 17. On the other hand, Reload Withotuntnn and ReloadW both exhibit better
45

---

Published as conference paper ICLR 2026
corrective accuracy performance than Reload but are still weaker than RELOAD and have lower ACCretain-
Accretasm (1)
CIFARIO
CIFARIO0
Poisoning
RELOAD
-2.84 42.56
-5.08
Reload Ashhuttssnt
-3.28 12.40
-5.24
11.85
-4.23 12.11
-5.34
11:72
Reload WithCnneeK
0.24 20.12
0.49
=0:35
Interclass Confusion (IC)
RELOAD
-2,70 11.00
-26.18
4:29.07
-3.66 +2.18
-16.83
+22.29
Reload Wnlnommationnn
11.50
-16.67 1:2.66
RelondWWatKCssneKK
0.01
0.24 =0:15
Num. Corrupted Samples
100
CIFAR 100
Poison
100
CIFARIO
500
CIFAR100
100
Figure 18: Model Utility across variants of RELOAD
C.3 ABLATION: RELOAD ON VISION TRANSFORMERS
We study the impact of layer normalization the performance of RELOAD We train Vision Trans- former (Dosovitskiy et al., 2020) on the CIFAR-10 dataset (Krizhevsky, 2012) and randomly unlearn 6000 data samples (10% of CIFAR-10).
We reuse PyTorch implementation a VIT (Wang et al., 2025) and train the model for 1000 epochs with learning rate le-4 using the Adam (Kingma & Ba, 2014) optimizer. The baseline model is trained to accuracy of 99.94%. Results are presented in Table 30.
Method
RA
AFA (I)
AFE (4)
APMIA (1)
RSKL (1)
FSKL (4)
RELOAD (OURS)
99. 99.45+0.12
0.53-0.50
0.79+0.1
0.01
0.19
8.7+0.11
Retrained (Baselinc)
99.91
51.11
4.31
0.31
Table 30: 10% Random Forgetting CIFAR-10 (VIT)
t: the goal to have as high a value as possible, A the value in the table the difference between result of the unlearning method and retraining (bottom row) on the metric and the goal is to have low difference, J: the goal is to have as low of a value possible. The bottom row presents the absolute value of M each metric. For any metric with A, the raw value is instead reported. Rows for AFA (J), AFE (J), and AFMIA (4) present the absolute difference in the value of the corresponding method this metric to the value of M(a- on the metrie. These results show that RELOAD performs similarly to the ground truth Retrained model on RA, AFA, AFMIA, and RSKI.. RELOAD strays from the Retrained model in AFE and FSKL..
C.4 ABLATION: RELOAD WITH QUANTIZED GRADIENTS
RELOAD incurs a storage overhead when caching gradients. We explore the feasibility of quantizing the cached gradients, to reduce the footprint of the algorithm. In this experiment, unlearn 6000 sam- ples (10%) CIFAR-10 from a trained ResNet-18 model and we quantize the cached gradients from torch. float32 torch. float16. 6. Before proceeding with unlearning, expand these gradients
46

---

Published as a conference paper ICLR 2026
back to torch. float The ResNet18 model is trained for 400 epochs with a learning rate of 1e-3 using the SGD optimizer. The model has trained accuracy of 99.76%. Results are presented in Table 31.
Method
RA (T)
AFA (4)
AFE (1)
AFMIA (4)
RSKL (1)
ESKL. (4)
RELOAD (UN-QUQ QUANTIZRD
90.99+0.0 01
0.467
0.76=0.00 0.0
0.01 +0.01
0.44
4.04+0o.1
RELOAD (QUANTIZED)
99. 00.00
0.46
0.7020.0s
0.01
0. 0.44
4.04
Retrained (Baseline)
99.52+0.16
36, 20.494
2.25=0.03
0.51+0.01
Table 31: 10% Random Forgetting on CIFAR-10 (ResNet-18) with Quantized Cached Gradients
t: the goal is to have high of a value possible, A+: the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and goal is to have low difference. the goal have as low a value as possible. The bottom row presents the absolute value M(a-> on each metric. For any metric with A, the raw value is instead reported. Rows for AFA (4), AFE (+), and AFMIA (4) present the absolute difference in the value of the corresponding method on this metric to the value of M(a-) on the metrie. These results show that RELOAD performs similarly the ground truth Retrained model RA, AFA, AFE AFMIA, RSKL. RELOAD strays from the Retrained model in FSKL.
As observed, performance deterioration across key unlearning and utility metries is minimal. Gradient quantization is thus highly effective and low-cost solution for mitigating storage overhead of RELOAD in models where full gradients are required.
C.5 ABLATION: RELOAD WITH FINETUNING ON OUT-OP-DDSTTIBUTION DATA
this section we explore the practicality of applying RELOAD after some finetuning has been performed on the model. Theoretically, significant parameter changes during finetuning could reduce the fidelity of the cached gradients relative to the current model state. As such, to operate in this setting, the cached gradients must be recomputed over the training and finetuning dataset after finetuning is completed. To explore this, we first train a ResNet18 8 model on CIFAR-10 for 400 epochs with a learning rate of 1e-3 to a training accuracy of 99.76%. Afterwards,
finetune the model the Recht et al., 201 88 dataset for 100 epochs with a learning rate of le-3. Prior to finetuning, the model had an accuracy of 30.8% on CIFAR1O.1. After finetuning, the model achieved an accuracy of 99.2% CIFARIO.1.
CIFAR-10.1 (Torralba et al., 2008;
We evaluate RELOAD unlearning in 3 modes. 1) We unlearn samples from the original CIFAR10 10 dataset after finetuning on CIFARIO.1 (Table 32, 2) We unlearn samples from CIFARIO.1 after finetuning on CIFARIO.J (Table 33), 3) We unlearn samples from both CIFARIO and CIFARIO.1 after finetuning on CIFARIO.1
(Table 34).
Methed
(f)
AFA (4)
(L)
AFMIA (1)
RJ)
FSKL
RELOAD (OURS)
98. 18.001
0.57+
0.06+0.03
0.01
0.91+0.02
3.45+
Retrained (Baseline)
92.34
34.57+0000
2.33
0.51
Table 32: 10% Random Forgetting from CIFAR-10 after finetuning on CIFAR-10.1 (ResNet-18)
Method
RA (1)
AFA (1)
AFE (4)
XFMIA (1)
RSKL (4)
FSKL (1)
RELOAD (OURS)
98.8910.18
2.85
0.20go.33
0.05
3.95
Retrained (Bascline)
99. 09.13
32.442.18
2.70.0.:1
0.5444..0
Table 33: 10% Random Forgetting from CIFAR-1001 after finetuning CIFAR-10.1 (ResNet-18)
47

---

Published as a conference paper ICLR 2026
Metbed
RA (f)
AFA (4)
(D)
APMIA (1)
RS)L
FSKL
RELOAD (OURS)
99. 08.69
3.88+1.62
0.27+0.11
0.02+0.01
1.04-0.18
3.97+0.15
Retrained (Bascline)
89.39 +9.39.22.0
35.83+0.95
2.3440.00
0.02-4.002
Table 34: 10% Random Forgetting from CIEAR-10 and CIEAR-10.1 after finetuning on CIFAR-10.1 (ResNet-18)
We observe that across 10 seeds, RELOAD exhibits stronger performance across key metrics with respect the retrain baseline on all three experiments. This suggests that the continual caching of gradients post finetuning proves RELOAD effective in scenarios where the base model is fine-tuned on distributionally shifted data.
An inherent requirement for this robustness is that gradients must be re-computed across the cumulative dataset (original + finetuning data) after sigmificant model updates. While this necessitates continual storage of the training set during the model's active lifecycle, argue this aligns with standard industry operations wherein data holders typically preserve their full datasets for versioning, debugging, and compliance until a specific deletion request is received.
C.6 ABLATION: FINETUNING RELOAD WITH SUBSETS OF Dretain
By design, Steps (1-4) of RELOAD necessitate the use of all of Dretain- Finetuning, Step (5), in literature does not always require the entire dataset. Here we ablate the usage of subset of Dretain during the finetuning step of RELOAD. We randomly select {10, 20, .... 90, 100 % of "Deetain use for Step (5) of RELOAD We observe that the performance of RELOAD on RA, AFA, and RSKL is directly correlated with the portion of D) retain used for finetuning, Interestingly, lower AFE and FSKL scores produced when finetuning with smaller portions of Dretain, despite larger AFA. These results are presented in Table 35.
Method
(1
AFA (4)
AFE (4)
AFMIA (4)
RSKL (4)
FSKL (4)
RELOAD (10% OF Drstass)
32.3310.33
11.44
0.1920.01
0.01
4.32
3.60
RELOAD (20% Dretain)
42.33+0..27
8.10=0.5a
0.26+0.04
0.01+
3.82+0.06
3.62
RELOAD (30% Dretats)
50.37
6.9220.74
0.2520.04
0.01
3.35
3.60
RELOAD (40% Dretats)
58.3010.35
5.53
0.34g0.05
0.01 01
2.96.0.00 06
3.75
RELOAD (50% Dictain)
65.7840..0 26
4.68=1.04
0.43
0.01 01
2.57
3.88
RELOAD (60% OF D Dretain)
72.86+4..22
4.00=0.78
0.42++0.05
0.01
2.1240.08
3.89
(70% Dretain)
79.91
3.1620.71
0.49
0.01
1.7240.02
4.01-2.1
RELOAD (80% OF D Destaco)
86.7610.14
2.6 12..8
0.54g0-05
0.01
1.31 1.31
4.10.
RELOAD (90% Dretain)
10
2.27=0.75
0.55
0.01
0.87+20.01
4.15
RELOAD (100% Drstein)
99.9910.01
1.6220.58
0.51g0.05
0.01
0.461:0.2
4.25
Retrained (Baseline)
99.6.20.07
36.57.10.58
2.24:20.3
0.5140.01
Table 35: 10% Random Forgetting CIFAR-1O (ResNet-18) Finetuning with different sized subsets of Dretain.
48

---

Published as a conference paper ICLR 2026
C.7 ABLATION: RELOAD OPERATES IN HUMAN SENSITIVE AND TABULAR CONTEXTS
Human- data domains are primary use case for unlearning in the real-world. In previous work (Newa- tia et al., 2024), we demonstrate that RELOAD successfully unlearns random samples and entire features from a TabNet model (Arik & Pfister, 2021) trained the Adult Census Income dataset (Kohavi, 1996; Asuncion et al., 2007), a standard benchmark involving sensitive personal attributes. Full results and discussions are presented in Newatia et al. (2024).
Forgetting Random 30% of the Data
Forgetting a Random Feature
Method
RA (1)
AFA (4)
AFE (4)
AFMIA (4)
Cost
OA (f)
RA (1)
TA (1)
Cost (4)
Original
N/A
NIA
V/A
NIA
NIA
0.84
0.73+0.m
0.8240.01
N/A
Retrain
0.84
0.82
0.48
0.51
1.00
0.77 +0.09
0. 0.84
0.70.g0.09
1.00
RELOAD
0.84
0.02+4.01
0.00
0.00
0.29 18
0.78 0.78gg.03
0.81 0.81 +0.01
0.77
0.40
GA
0..2
0.04
0.5540.04
0.06=0.07
0.35+ 0.13
0.48+a.11
0.35
0.30
FT
0.001
0.01
0.0110.00
0.05
0.53
0.77 10.00
0.84
0.76.1.08
0.7910..0
Table 36: Results highlighting the performance of RELOAD on the tabular Census Income dataset.
C.8 ABLATION: LEARNING RATE T/p AND THRESHOLD a
We study the effect of different learning rates the unlearning performance exhibited by the RELOAD algorithm. For this study, we select the case of randomly forgetting 10% of the training data from a ResNet-18 model trained CIFAR-100.
NA
A AFA
AFE
AFMIA
1
82
35
14
AAUC
Cost
NSKL
FSKL
Figure 19: Impact of Learning Rate (1) on RELOAD performance.
As shown in Figure 19, we observe that the choice of learning rate has significant impact performance. This is particularly true in the case of AFA, AFF, AFMIA, and AUC measurements which are the primary metrics evaluating how well the model has forgotten D forget- Based on these plots, choose 7) 0.33.
Figure 20 shows the effect of varying the proportion of the parameters that are selected for reinitialization (a). We observe that the choice of threshold has impact on the performance of the RELOAD algorithm and that
49

---

Published as conference paper ICLR 2026
NA
AFA
AFE
AFMIA
540
560
AAUC
Cost
NSKL
FSKL
Figure 20: Impact of Threshold (a) on RELOAD performance
its selection involves tradeoff between the different metrics we consider. Thus, the best choice of should ideally be selected through a hyperparameter search.
50


Reviews:
[{"review_id": "6wwJ3O2oIi", "reviewer": "Reviewer_VSdc", "summary": "This paper addresses a significant limitation in existing machine unlearning techniques, namely the requirement to retain access to the “forget set” (the data to be removed) during the unlearning process. The authors introduce the concept of partially-blind unlearning (PBU) and propose RELOAD, a framework that performs unlearning without direct access to the forget set. RELOAD leverages cached gradients from the final epoch of training and combines gradient ascent, selective weight reinitialization, and fine-tuning on retained data. The paper claims strong empirical results showing that RELOAD can efficiently approximate retraining from scratch and even outperform some forget set-dependent approaches, including applications to both image classification and large language models.", "questions": "1. How would RELOAD perform if the model undergoes fine-tuning or incremental updates after initial training?\n\n2. What is the approximate storage cost of ∇θL(D) for large models such as Llama2-7B, and could this be mitigated via gradient compression?\n\n3. Have the authors tested for information leakage from cached gradients using existing gradient inversion techniques?\n\n4. How does RELOAD handle overlapping or correlated data between forget and retain sets?\n\n5. Can the method scale to federated or distributed training settings where only local gradients are available?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The problem formulation of unlearning without the forget set is timely, novel, and practically relevant for privacy compliance (e.g., GDPR).\n\n* Methodologically sound integration of gradient ascent, weight reinitialization, and fine-tuning into a coherent framework.\n\n* Empirical results show promising efficiency and competitive performance, especially for large models such as Llama2-7B.", "weaknesses": "* Limited robustness to model updates\n\nThe method assumes availability of final-epoch gradients representing the entire training data. In real-world pipelines where models are fine-tuned or continuously updated, these cached gradients may no longer capture the forget set’s influence, reducing unlearning effectiveness. Evaluating RELOAD under fine-tuning or continual learning scenarios is necessary.\n\n* Unclear privacy guarantees\n\nAlthough RELOAD is “partially blind,” cached gradients can still leak sensitive information. Prior works (e.g., Geiping et al., 2020) show that gradient inversion can reconstruct data. The paper lacks a quantitative privacy leakage analysis to support its safety claims.\n\n* Limited ablation and sensitivity study\n\nWhile some ablations are included, deeper exploration of how hyperparameters (e.g., ascent rate, reset proportion) affect performance is missing. This limits confidence in robustness across settings.\n\n* Storage overhead\n\nThe approach removes the need to store the forget set but introduces the requirement to store full-model gradients, which can be large for modern networks. The practical feasibility of gradient caching is not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JI5GkT2Ur3", "reviewer": "Reviewer_WbA5", "summary": "The paper proposes RELOAD, a partially-blind unlearning (PBU) method that aims to remove the influence of a forget set without access to the forget data. The method includes three parts: 1) an ascent step using cached final-training gradients, 2) re-initialisation of parameters with low “knowledge value,” and 3) fine-tuning on the retain set to maintain model utility. Experiments include classic unlearning, entity unlearning, and corrective unlearning.", "questions": "Please see the Weaknesses section for all questions and clarification requests.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of forget-set-free unlearning is novel and well-motivated. The RELOAD enables machine unlearning without retaining the raw forget set by using cached final-step gradients.\n\n2. The empirical results are strong across benchmarks. RELOAD can preserve model utility while improving forget quality.\n \n3. The method is efficient. For Llama-2-7B, it uses 7% of weights and <0.025% of retained data, finishing in 8 minutes on a single GPU. This result suggests that the method is practical.", "weaknesses": "1.\tThe paper does not provide a theoretical justification for why a single gradient-ascent update using $\\nabla_\\theta L(D)-\\nabla_\\theta L(D_{\\text{retain}})$ and selective re-initialization of low-KV weights can remove the influence of the forget data. As a result, it is unclear when RELOAD succeeds or fails beyond the reported scenarios.\n\n2.\tThe paper claims RELOAD “allow user data to be immediately removed when a request for deletion is made, eliminating the continued accumulation of dataset risk,” but RELOAD requires the retain set to compute gradients and to perform fine-tuning. This means that the retained data is still at risk, and the ideal method should not use any training data during the unlearning process.\n\n3.\tThe robustness of RELOAD across different numbers of requests is unclear. When the forget set is extremely small (e.g., unlearning only one data point), $\\nabla_\\theta L(D_{\\text{forget}})=\\nabla_\\theta L(D)-\\nabla_\\theta L(D_{\\text{retain}})$ becomes a very small residual between two nearly identical large gradients; this makes the ascent step near zero and may fail to truly forget that sample. In addition, when the forget request is large (e.g., 30% of the data), the ascent and re-initialization would degrade model utility, while fine-tuning may fail to recover the utility given the limited retain set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qH9vk78KHd", "reviewer": "Reviewer_2ZVC", "summary": "This paper tackles the problem of machine unlearning without retaining the forget set, a long-standing practical limitation in enforcing the “right to be forgotten.” The authors introduce RELOAD, a partially-blind unlearning algorithm that relies only on cached gradients from the final training step and the retained dataset, avoiding the need to store sensitive data. Through gradient ascent, selective reinitialization, and fine-tuning, RELOAD achieves performance comparable to retraining from scratch while being significantly more efficient. Experiments on image classification and large language models demonstrate promising results.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper formalises the partially-blind unlearning setting, which is a realistic and privacy-preserving variant of traditional unlearning.\nThis addresses an important gap between regulatory demands (e.g., GDPR) and existing technical capabilities.\n2. The motivation connecting dataset risk and model risk is conceptually strong and provides a clear societal justification for this work.\n3. RELOAD elegantly combines gradient-based unlearning, structured sparsity, and fine-tuning in a simple yet effective pipeline.\nThe “knowledge value” mechanism for selective reinitialization is particularly interesting.\n4. Across diverse tasks (CIFAR, SVHN, and Llama-2), RELOAD shows comparable or better performance than methods requiring the actual forget set. Its efficiency (<8 min for Llama2-7B) suggests real practical potential.", "weaknesses": "1. Limited theoretical justification. The paper mainly relies on intuition and empirical validation. While the derivation of ∇L(Dforget) = ∇L(D) − ∇L(Dretain) is sound, the guarantees of approximate unlearning (e.g., bounds on residual influence) are not formally analysed.\n2. Dependence on cached gradients. Storing full-model gradients at the end of training may be expensive for large-scale models, potentially offsetting some of the claimed efficiency or privacy advantages.\n3. While results are impressive, the experiments could be broadened — e.g., include more realistic privacy benchmarks or human-sensitive data domains.\n4. Some ablation studies (e.g., varying α in reinitialization, or using partial gradient caching) are deferred to the appendix but would strengthen the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXeuBEr62a", "reviewer": "Reviewer_dom5", "summary": "Existing study requires the model provider to keep the requested unlearned data information until the unlearning is completed. This paper explores and defines a new unlearning setting, Partially Blind Unlearning (PBU), in which no direct access to the unlearning data is required. Under this setting, the author implements a three-fold method that leverages cached gradients from the training stage. The method RELOAD consists of three steps, combining previous studies. RELOAD gives the unlearned model under PBU by following 1. Compute gradient difference, perform a single gradient ascent step, and then fine-tune on retained datasets. The author then provides comprehensive experiments on both classical unlearning tasks unlearning on vision models, and unlearning tasks on language models.", "questions": "It seems like RELOAD is achieving good results in the baseline section. The setting for the experiments is not strict to PBU. This indicates that  RELOAD is producing better results and outperforms baseline methods that do use the forget set. If so, why is PBU important to this method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a new privacy-oriented perspective on the problem of unlearning. The author made an important observation and defined a new setting for exploring safe unlearning. The motivation is well-explained. \n2. The paper integrates several existing methods in solving new and practically important problems. This provides a more modular and interpretable unlearning algorithm. \n3. The evaluation is comprehensive and representative. The experiments marked the performance of RELOAD on both small-scale vision models and on language models.", "weaknesses": "1. In Section 2.3, the author states that we can infer the loss gradient on the forget dataset with the original model by computing the difference between the loss gradient on the original dataset and the retained dataset. This, however, works with several assumptions that are not explicitly stated, such as the assumption that loss is additively computed across samples. I am concerned that for contrastive learning tasks or models trained with layer normalization, this formula may not function as intended. I would appreciate further discussion on this point.\n2. While the overall unlearning method provides more interpretability compared to existing unlearning methods and achieves promising results, the methodological innovation is somewhat incremental. The core part of the method, including gradient ascent, selective weight re-initialization, and fine-tuning are existing method. The contribution of the method lies mostly in how to combine the methods under PBU setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)