FaLW: A Forgetting-aware Loss Reweighting for Long-tailed Unlearning
Authors: Liheng Yu
Zhe Zhao
Yuxuan Wang
Pengkun Wang
Xiaofeng Cao
Binwu Wang
Yang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Long-tailed learning
Unlearning
Fairness
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Machine unlearning, which aims to efficiently remove the influence of specific data from trained models, is crucial for upholding data privacy regulations like the ``right to be forgotten". However, existing research predominantly evaluates unlearning methods on relatively balanced forget sets. This overlooks a common real-world scenario where data to be forgotten, such as a user's activity records, follows a long-tailed distribution. Our work is the first to investigate this critical research gap. We find that in such long-tailed settings, existing methods suffer from two key issues: Heterogeneous Unlearning Deviation and Skewed Unlearning Deviation. To address these challenges, we propose FaLW, a plug-and-play, instance-wise dynamic loss reweighting method. FaLW innovatively assesses the unlearning state of each sample by comparing its predictive probability to the distribution of unseen data from the same class. Based on this, it uses a forgetting-aware reweighting scheme, modulated by a balancing factor, to adaptively adjust the unlearning intensity for each sample. Extensive experiments demonstrate that FaLW achieves superior performance.

Published as a conference paper at ICLR 2026
FALW: A FORGETTING-AWARE Loss REWEIGHTING FOR LONG-TAILED UNLEARNING
Liheng Yu', Zhe Zhao" Yuxuan Wang', Pengkun Wang" 12; Xiaofeng Cao Binwu Wang Yang Wang 1.2-
'University of Science and Technology of China(USTC)
Suzhou Institute for Advanced Research, USTC *City University of Hong Kong
*Tongji University
edu. cn, xiao cao. ut st com, pengkun@ustc.edu.cm
{yuliheng, shnng, zz4543, xuco,
ABSTRACT
Machine unlearning, which aims to efficiently remove the influence of specific data from trained models, is crucial for upholding data privacy regulations like the "right to be forgotten". However, existing research predominantly evaluates unlearning methods on relatively balanced forget sets. This overlooks a common real-world scenario where data to be forgotten, such as user's 's activity records, follows long-tailed distribution. Our work is the first investigate this critical research gap. We find that in such long-tailed settings, existing methods suffer from two key issues: Heterogeneous Unlearning Deviation and Skewed Unlearn- ing Deviation, To address these challenges, propose FaLW, plug-and-play instance-wise dynamic loss reweighting method. FaLW innovatively assesses the unlearning state of each sample by comparing its predictive probability to the dis- tribution of unseen data from the same class. Based on this, uses a forgetting- aware reweighting scheme, modulated by balancing factor, to adaptively adjust the unlearning intensity for each sample. Extensive experiments demonstrate that Fal.W achieves superior performance. Code is available at Supplementary Ma- terial.
INTRODUCTION
With the enactment of regulations such the General Data Protection Regulation (GDPR), the "right to be forgotten" [Regulation] 2018: Hoofnagle et al., 2019) has become a cornerstone of data privacy protection. In artificial intelligence, the right's realization has given rise to emerging technical paradigms of machine unlearning. Its core objective is to accurately and efficiently remove the influence of specific training samples from a pre-trained ML model (Shaik et 2024
Currently, machine unlearning methods can be broadly categorized into two main types: exact un- learning (Guo et al.) 2019) and approximate unlearning (Izzo et al., (22II. The former primarily focuses methods that provide provable guarantees or certified removal. Within this category, re- training from scratch considered the gold standard (Thudi al., 20226). This approach involves training a new model from scratch on the retain set, which is the original dataset minus the data to be forgotten. However, such retraining-
approaches are computationally prohibitive and thus impractical for large-scale models prevalent today.
In contrast, approximate unlearning presents a practical alternative, aiming for fast and effective removal of data influence. While these methods may not always come with formal guarantees, their efficacy is typically evaluated through empirical metrics, such as Membership Inference Attacks (MIA) (Carlini et al. 2022), and they often operate under fewer constraints the data, model, or algorithm compared to their exact counterparts. To date, the evaluation of machine unlearning has
"Corresponding author.

---

Published as a conference paper at ICLR 2026
predominantly centered image classification tasks, where, typically, the forget set is constructed by randomly sampling a subset of the original training data.
However, empirically observe (Fig-
ure aaa) that random sampling consis- tently yields forget sets that nearly balanced. This stands in stark contrast many real-world scenarios, where for- get requests are often naturally long- tailed. For instance, when user deacti- vates their online shopping account (Bal- asubramanian et al.] 2024). their data, shaped by personal interests, is typically concentrated in few specific categories. Based on this, machine unlearning un- der such long-tailed distributions remains largely unexplored. To bridge this gap, we introduce the concept of Unlearning De- viation quantify the unlearning outcome for individual samples. Through analysis on imbalanced forget sets, we uncover two critical phe- nomena:
(a)
500
Long tailed
(b)
Balanced
Low
Seed
10
200
Sanad
Seed
Density
200
More
High
100
SalUn
Retrain
0.4
Class Index
Prediction Probability
Figure I: (a) Distributions of 20% CIFAR-100 for- get set generated via balanced, long-tailed, and ran- dom sampling (Seeds 1-3 denote different runs). (b) Predicted probability distributions for SalUn vs. Re- train (ResNet-188 18) after unlearning 20% randomly sampled forget set from CIFAR-100.
Heterogeneous Unlearning Deviation: models tend to over-forget some samples while under- forgetting others.
Skewed Unlearning Deviation: the magnitude of this deviation is disproportionately severe for samples.
Existing methods, designed with a holistic perspective, are ill-equipped handle instance-level de- viations arising from long-tailed forget distributions. Therefore, an instance-leve unlearning method in urgent need, and similar ideas have also been reflected in (Zhao al., (2024).
To address this challenge, we propose FaLW, a plug-and-play instance-wise dynamic loss reweight- ing method. Our key innovation is to estimate the unlearning deviation for each sample by mea- suring its predictive probability against the distribution of probabilities for unseen data of the same class. Based on this, we design forgetting aware dynamic weight. Furthermore, we intro- duce a balancing factor to modulate the sensitivity of these weights, making the adjustment more ag- gressive for tail-class samples. FaLW dynamically assesses the unlearning state of each instance and adjusts its unlearning intensity accordingly, effectively mitigating both heterogeneous and skewed unlcarning deviations. In summary, our main contributions are as follows:
Brand-new Perspective: This the first work to investigate the long-tailed unlearning prob- lem and formalize the phenomena of Skewed and Heterogeneous Unlearning Deviation, filling a key gap in the current field.
We propose to utilize the predicted probability distribution over un- seen data assess the instance-wise unlearning state during the process. Based this, we introduce FaLW, forgetting-awan aware dynamic loss reweighting that effectively mitigates the identified unlearning deviations in long-tailed scenarios.
Plug-and-play Solution:
Superior Effect: Extensive experiments on multiple image classification benchmarks demon- strate that our method achieves state-of-the-ar oh-the-rrt performance.
RELATED WORK
Machine Unlearning (MU). The field of Machine Unlearning (MU) has emerged in response to the growing need to selectively remove the influence of specific data from previously trained models (Shaik et al. 2024 Xu et al.) 2024 Neel et al. 2021 Sekhari et al., (021). Methodologies for exact MU, which offer provable guarantees of data removal, have been thoroughly explored, especially for
linear and convex models (Guo et al.) 2019; Koh & Liang 2017: Giordano et al.) 2019). However, applying exact unlearning to deep learning models typically necessitates complete retraining from
2

---

Published as a conference paper at ICLR 2026
scratch (Bourtoule et al. 2021). procedure that is computationally prohibitive and thus impractical for a majority of wealdworld applications. To enhance efficiency, an alternative approach. termed
approximate MU (Xu et al.
2024 Thudi et al. 2022a), has gained prominence. A multitude of
gradient-based approximate un unlearning techniques (Golatkar et al. 2020; Warnecke et al. 2021 Jia
et al., 2023 Graves et al. 2021 Thudi al. 2022a Chundawat et al. [2023] [ et al. [2024]]] Fan
et al. 2024, Huang et al., 2024) have been introduced, which utilize specialized loss functions to
guide the model erasing information related to designated data, consequently mitigating privacy
vulnerabilities like membership inference attacks (Hu et al. [022 Li & Zhang 2021; Song et al.,
2019 Carlini al. 2022) and data reconstruction attacks (Fredrikson et 20.55 Balle et al. 2022). A significant challenge, akin to catastrophic forgetting in continual learning (McCloskey & Cohen
1989 Wang et al. 2024 Ratclifi, 1990), arises when approximate methods focus excessively on the forget set, often causing substantial decline in model performance on the remaining data. While various strategies (Tarun ct al., 2023 Kurmanji et al. 2023 Fan et al., 2024; Huang et al. 2024), notably fine-tuning on the retain set, have been introduced preserve model utility, assert that a critical real-world condition has been overlooked. Real-world unlearning data often follows long- tailed distributions, yet existing methods remain unexplored in long-tailed MU regimes. Our FaLW fills this gap, enhancing long-tailed approximate unlearning performance.
Long-tailed Learning (LTL).. Existing L.TL methods can be broadly classified into re-sampling and re-weighting, transfer learning and knowledge distillation, and multi-expert or modular de- signs. Specifically, re-sampling(Chan CChawla et al. 2002 He et al., 2008) and re-weighting(Cur et al. 2019 Cao et al. 2019) techniques balance the data distribution or the learning process by adjust- ing sample probabilities or loss weights. Transfer learning (ii
al. 2019 Liu et al. 2019) and et al. 2020 He et al., (02T)) methods attempt to transfer knowledge from head classes to tail classes or leverage knowledge from large pre-trained models. However, the long-tailed problem in MU is distinct from traditional LTL. The distinction lies in the objective: while traditional methods aim to improve performance on tail classes, long-tailed unlearning must effectively remove information from imbalanced set while simultancously accounting for the un- learning performance changes that long-tailed data induce on both head and tail classes. Therefore, developing imbalance-ane methods specifically tailored for the machine unlearning paradigm is urgently needed. In this paper, motivated by our findings in the long-tailed unlearning scenario, we design forgetting-aware approximate unlearning method suitable for this challenging setting.
knowledge distillation(Xian)
3 PRELIMINARIES
Notation. We consider probability distribution Fx over the input space X' and set of C classes denoted by y = {1, 2..,,....C', C}. We focus on a multi-class classifier F X + V. For a given input E X, the model's prediction function, fxx), outputs a vector of predicted probabilities p(-1z) E RC over the classes. The loss function for the model F is denoted by Ey- : X x ) R4, which quantifies the discrepancy between the predicted probabilities and the ground-truth label In the supervised learning setting, we are given a dataset DD = {(a,, W:) consisting of N labeled samples where each input x; is drawn from 1xxx and y/2 y is its corresponding label. The model F. parameterized by 0, is trained on D by minimizing the empirical risk, often expressed as E (( v)). The resulting set of parameters from training denoted by 0..
Machine Unlearning, Let O, denote the parameters of the initial model trained on the entire dataset D. Given subset D, C D to be forgotten, its complement, the retain set, is defined as D, :>=DD,,, D),, The most straightforward unlearning method is retraining from scratch, which trains model exclusively on the retain set D, to obtain the parameters 0. = arg min, E, [(x(1, y)]- Typically, retraining is here considered the gold standard for unlearning. However, retraining is often computationally prohibitive.
practical approach is approximate unlearning, which applies an algorithm M to update the original parameters O, by removing the influence of Dy. The resulting unlearned model parameters are denoted by 0, = M(,,, [D,, DzJ). The objective is to ensure that the behavior of the unlearned model with parameters 0., closely approximates that of the gold- standard retrained model with parameters 0..
Long-tailed Settings. N,, denote the number of samples for class k. A long-tailed distribution refers to a highly imbalanced data distribution. The sample distribution can often be modeled by a power law, such as Age k_?, where the imbalance factor y controls the degree of class imbalance.
3

---

Published as a conference paper at ICLR 2026
Head
Mid
Tail
Retrain
Accuracy Gap %
Accuracy Gap %
Retraln
Accuracy Gap %
-5
10
Retrain
-10
-20
-2
-15
SalUn
SalUn
SalUn
-30
Ours
-20
Ours
Ours
1/3
1/2
3/2
1/4
1/2
2
1/4
1/3
1 3/2 Y
Y
Y
Figure 2: Under the setup of the ResNet-18 architecture trained CIFAR-100 dataset with 30 30% unlearning ratio and a forget set distribution defined by N₁. x E7. this figure visualizes forgetting accuracy gaps between Salun, our method (Ours), and the Retrain baseline across head, mid, and tail classes within the forget set.
For notational convenience, class within the forget set D and the retain set DD, respectively. Their cardinalities are given by Ny.x yP and N |D,, We define the long-tailed unlearning problem the scenario where the forget set Dy itself exhibits a long-tailed distribution, satisfying Wf 2 NN 2 Nf an. >> NN,c. Following standard practice long-tailed learning, we partition the classes into head, medium, and tail groups based on the number of samples per class in the forget set, such that the total number of samples in each group satisfies Ny N f.medium > N f
DJA and Dr. k to denote the subsets of samples belonging to
4 MOTIVATION
Definition (Unlearning Deviation). For any sample (I,, Mc) Dj and a small positive threshold Ts we categorize the unlearning outcome for x, by comparing the output probability of the approx- imate model On with that of the retrained model O, on the ground-truth class Vt- The outcome
classified as:
Under-forgetting, forgettn. Over-forgeting, if Po.. (c(xz) Po. (c)a;) - -ii
if Po.. (c/x²) Po. (c|x:; +
if lpo. (c|z;) pe, (c/x;)] S Fi
Here, po(clzz) denotes the probability assigned to the true class e for input 2, by model with parameters 0. The conditions of under-forgetting and over-forgetting are collectively termed Un- learning Deviation.
Formulation 1 (Gradient-basee Approximate Unlearning). Briefly, gradient- based approximate un- learning methods can be formally expressed in three parts:
min l unieear (0.)) 0
8 L(Dr:O.)
A- R(00. 0.)
(2)
Forgetting Execution
Capacity Preservation
Parameter Construints
In Formulation [ Forgetting Execution involves computing the inverse gradient derived from D,, which is essential for implementing the model's forgetting process. @ Capacity Preservation leverages D, to preserve the model's s predictive accuracy on the remaining data. This is achieved by either constraining the inverse gradient (e.g., via projection methods) or introducing forward gra- dient during optimization. @ Parameter Constraints term, R(-), imposes constraints on the updated parameters 0,, for instance, by penalizing large deviations from the original parameters O, by en- forcing properties like LI Ll-sparsity. The coefficients a, 3 and are non-negative hyperparameters that balance the trade-offs between these competing terms.
When an approximate unlearning method has access only to the forget set D,, is prone to severe over perrfoorettting Formally, for an unlearned model 6,, MM0,, Dy), the condition for over-forgetting, po., (c(x:) po. (c(x.2) Ts, frequently met for samples (xt, e) E D,,
Proposition 1.
4

---

Published as a conference paper at ICLR 2026
The proof for Proposition[] is deferred to Appendix A.1 ful unlearning does not require the model's confidence on a forgotten sample to be very low and even driven to zero. When the retain set D, is unavailable, the extent of unlearning cannot be effectively calibrated, and the process lacks a natural stopping point, leading to over-forgetting. Consequently, most approximate unlearning methods leverage both D and D,,, utilizing the forward gradient on D, (the capacity preservation term) to counteract the aggressive parameter updates from the forget- ting term. To further investigate this interplay, we conduct experiments on SalUn (Fan al. 2024). which embodies all components of the objective in Formulation |
This proposition underscores that success-
As illustrated in Figure (bb and Appendix B.1 the predictive probabilities of the unlearned model O produced by SalUn are consistently lower than those of the retrained model 0. across various experimental settings. This indicates that despite leveraging the retain set for capacity preservation, the SalUn method still exhibits a discernible degree of over-forgetting.
Observation
Random Sampling is more Balanced. In many practical unlearning scenarios, the data desig- nated for forgetting is inherently imbalanced. For instance, when a user requests the deletion of their account, a platform must ensure this user's data no longer influences its models (e.g.. for recommen- dation or advertising). An individual's data, shaped by personal interests, is often highly skewed and concentrated within a specific subset of categories. However, prevailing research has largely simulated forget requests by randomly sampling from the training data. While random sampling could theoretically yield an imbalanced forget set, our empirical results indicate it consistently pro- duces class distribution that is far more balanced than long- long-tailed one. Specifically, shown in Figure [a), repeatedly sampling 20% of the CIFAR- 100 dataset results in distributions that closely approximate a balanced one, starkly contrasting with the characteristics of long-tailed distribution. This discrepancy highlights a critical gap: the behavior of approximate unlearning methods under the more realistic long-tailed forgetting scenario remains insufficiently explored.
When Unlearning Meets Imbalance. To address this gap, we conduct extensive experiments on CIFAR-10M 100, with details presented in Figure 2 and Appendix [B.2] We construct long-tailed forget sets where the class distribution follows NN с k with the exponent controlling the severity of the imbalance. As we increase % observe a critical trend: the unlearning outcome for head- class samples gradually shifts from over-forge to under-forgetting while the under-forgetting for medium- and tail-class samples is exacerbated. We summarize these findings as follows:
Observation 2. Under a tnig-tailed/ieee forget distribution, approximate unlearning methods exhibit two key essences:
The model demonstrates disparate unlearning pat- terns across classes, typically under-forgetting samples from head classes while over- forgetting those from tail classes.
Heterogeneous Unlearning Deviation:
Skewed Unlearning Deviation: The magnitude of the unlearning deviation is disproportion- ately larger for tail-class data compared to head- and medium-class data.
Existing approximate unlearning methods are predominantly designed with a holistic perspective, focusing on the aggregate forgetting effect. Consequently, there is a dearth of research on strategies to handle the differential unlearning behaviors that emerge across samples. This limitation becomes particularly acute when the forget set exhibits long-tailed characteristics. Therefore, an adaptive unlearning mechanism that can modulate the process at finer, sample- or class-level granularity is urgently needed.
FALW: A FORGETTING-AWARD Loss REWEIGHTING
Motivated by the challenges identified in Observation we propose instance-wise loss weighting method. The overall architecture is depicted in Figure 3 This approach dynamically adjusts the loss weight for each sample based its estimated unlearning deviation during the un- learning process, thereby steering the model towards more faithful unlearning state. Specifically, introduce dynamic weight w; for each sample in D, and adapt the general objective from Formulation as follows:
min l unlcamn (.,))
Σ (DDD)))DD
wi (.-L((x,.)))) 0w) + 8. C,D, 0w) + A R(O..00) 0.)
(3)
5

---

Published as a conference paper at ICLR 2026
Asscssment of Uniesrning bevistion
F
d.
Uniearning
8.
COO
COO
Prodetee probed byy
M
Stop Fargettngg
right
Foggtting Under-farget
Forgetiig
Over-forget
Forthfllffrget
urlcarm
un lelm
Forgehtiggcoo caare Weight
Belance Facter
Dr 8,
Nagettve deviatio on
Mi sign(s.) tonh (abof())))a)))) d
Diraks
Heal
Contrn of
High weight
Mad
Over Forget
MMd chanlitg
Shory shongng
wiight
Prodicioon Fndtveiilyy
Prediction probability distribution unseen data
Unseen data
Lenient
Sensthive
Figure 3: The FaLW framework. Left: Analyzing instance-level unlcarning bias using the model's predictive probability on unseen data. Right: The top shows instance-level loss reweighting, and the bottom explains the forgetting-awee awaere weight and balance factor.
5.1 ASSESSMENT OF UNLEARNING DEVIATION
A Key Idea:
The core principle of unlearning is to transition a model's state from having been trained on a sample to a state equivalent to never having seen it, briefly as from "seen" to "un- seen". Essentially, unlearning a sample aims to nullify its marginal contribution to the model's parameters, thereby reverting the model's 's predictive behavior on that sample to its naive state. To formalize this, let's consider unlearning single sample (Ii, Vi = c). The objective is to update the model parameters from the original 0,, to the unlearned 0,, such that the model's confidence on the true class approaches specific target: pe,, (c|2z, + pe. (c/x;). Here, pe, (c/2,) represents the ideal predictive probability on z, from a model that has never been exposed to this sample. A way to get this ideal model, parameterized by 0., is retraining from scratch on the dataset with that specific sample removed: 0, = arg ming B(z,))DD P(e.w)) ((m))))) (2, y)].
Proposition 2. All else being equal, model trained on dataset including a sample (æi, Vi c) will exhibit higher confidence on that sample's 's true class compared to a model trained on the same dataset excluding it. Formally, let O, be the parameters of model trained on D, and let O, be the parameters of a model trained on 'D ((1,,,)). Then, generally holds that:
v.).
pe. (clz,) 2 Po. (c/z.)
(4)
Remark. The proof for Proposition 2 is provided in Appendix A.2 This proposition implies that
as long as knowledge of a sample x; is retained, the model's predictive confidence will be inflated relative to the ideal unlearned state. If we assume for moment that the target probability pe, (clx,) is accessible, Proposition inspires a conceptual model for an ideal unlcarning trajectory. Before the process begins, with parameters 0 0.. the model's 's confidence is high: po (o(c));) 2 po, (c/ar.- As unlearning unfolds, Po (c(2z; should monotonically decrease, eventually converging to the target pe. (c(x.). At this point, the sample successfully forgotten, and the process should terminate to prevent worgttorgt Therefore, during approximate unlearning, the discrepancy between the current confidence po_ (clx,) and the target po_ (c/x;, can serve as a metric to gauge the extent of forgetting for sample 24, provided the target is known.
2
Approximating pe. (c/x)). In practice, obtaining the exact target probability pe, (cla;) in- tractable during approximate unlearning. We therefore approximating the deterministic target into a target distribution. Specifically, we postulate that for a forgotten sample (x;, V/ = c). ideal predictive probability Po. (cl2x) should be indistinguishable from the probabilities of other samples of class e that the model has never seen. Let P. be a distribution of unseen samples of class e. The ideal unlearning process should result in po. (clz:) being plausible draw from the distribution {pe. [a' ~ }. As previously argued, unlearning aims to revert a sample's s status from "seen" genuinely unseen data for class represents the desired endpoint. Crucially, while the target probability for any single sample is elusive, this target distribution can be estimated during the unlearning process.
to "unseen". This distribution of probabilities

---

Published as a conference paper at ICLR 2026
Assessment of Unlearning Deviation. To operationalize this concept, model the distribution of predictive probabilities on unseen data for a given class с with a Gaussian, i.., Po po(c|2z) ~ (Hee (() for any unseen sample z' ~ P The Unlearning Deviation for specific sample (2;, Vi c) can then be dynamically assessed by measuring how its current probability po (c(orr) deviates from this target distribution. This dynamic assessment guides the unlearning trajectory, depicted in Figure 3 left. Initially, at 0 = O.. the model's confidence pa is significantly higher than Pes positioning it as a positive outlier of the distribution. This signifies state of under-forgetting. As the unlearning process progresses, pe (e(c)); decreases. When its value falls within high-density region of the distribution, we consider the sample faithfully forgotten, as its predictive signature is now typical of an unseen sample. At this stage, the unlearning pressure on this sample should be attenuated. If the process continues aggressively, po(clz;)) may drop substantially below pess becoming negative outlier and indicating a state of over-forgetting.
5.2 FORGETTING-AWARE WEIGHT
Based on the assessment method above, we introduce our Forgetting-Awar Weight wy for each sample (Mit, Ui = <) Dj- The weight is formulated
Wc = 1 + sign(z;) (tanh()=[))* with 2, Pt - 1e
(5)
dc
where pt = (clla)) is the model's current predictive probability on the sample. Here, z; is the standard z-score, quantifying how many standard deviations the current probability is from the mean He of the target distribution for class The sign(-) function determines the direction of adjustment, while the hyperbolic tangent, tanh(-), maps the magnitude of the deviation into the range (-, The hyperparameter 1j > acts temperature, controlling the sensitivity of the reweighting scheme; a larger 17 leads to a more pronounced response to the deviation, causing w; to change more sharply. In practice, the class-wise be and a, are estimated by running the current model on held-out validation set and calculating the mean and standard deviation of predictions for each class.
Remark. As depicted in the Figure 3 right, the reweighting scheme provides intuitive, adaptive control mechanism. When a sample x, is over-forgotten, its confidence p4 falls far below the mean Me. resulting in large negative z-score 2, Consequently, sign(2) becomes -1 and w, approaches zero (1 = 0), effectively halting the forgetting pressure on this sample. Conversely, for an forgrtfor sample, p; is positive outlier, yielding large positive 2; and weight w; that approaches 2 (1 + = 2), thereby intensifying the unlearning process. This adaptive mechanism is particularly crucial for addressing the Heterogeneous Unlearning Deviation observed in long-tailed settings, as it holistically mitigates the deviation for all forget samples by preventing both over- and fnder-forgetting.
5.3 BALANCE FACTOR
To counteract the Skewed Unlearning Deviation, where tail-class samples suffer from more severe deviation, we introduce a class-aware balance factor B, to make the reweighting scheme more sen- sitive for minority classes. We want the weight adjustment to be more aggressive for tail-class samples; for a given deviation, their weights should change more rapidly than those of head-class samples. To this end, define the balance factor as:
B, ( Ng CC.NAA)
(6)
where = yi. .N, |Dyl is the total number of samples in the forget set, f... is the number of samples of class e in the forget set, C is the total number of classes, and 7 is a non-negative hyperparameter. This factor is inversely proportional to the class frequency, yielding larger value for tail-class samples. We then integrate this factor into our forgetting -aware weight by modulating the exponent:
1 + sign(=:) (lanh(1=())
With this modification. for a tail-class sample with a large B, This causes weight w; to react more sensitively to z-score 2;, thus enabling more rapid response to its unlearning deviation. By
7

---

Published as a conference paper at ICLR 2026
Thny-Imagren Vig- 1gg-66.0 1
CIFAR-10, Vgr-16 2 1/2
Method
Kandom Forget (10%)
Randum Forget (89%)
Aug. Cap
Arg.
Retnon
25.65
99.98 (0.00)
48.81 (0.00)
14.36 (0.10)
0.00
65. (0/00)
100 10
78.07 (0.00)
34 05
0.00
FT
86.07 (61.33)
59.98 (0.20)
53.95 (5.12)
13.00 (61.33)
31.95
2.65
68.42 (2.48)
96.51 (3.48)
78.57 (0.25)
31.55 (2.48)
2.15
0.39
RL.
18.50 (3 13)
99.33 (0.15)
08.51 (0.32)
$1.30.0.14)
3.00
0154
72.57 16.43)
98.87
81.15 (2.53)
27 (6.43)
4.13
0.59
GIA
77. (50 14;
03:23 (6-26)
08.67 (0:15)
23.00 (50-14)
27:27
2.06
18.67 (7.27)
64.28 (15:713
67:21 (13.453
41.32(2.27))
10.43
3.00
IU
96.29 (70.66)
95.46 (4.52)
55.71 (6.38)
3.71 (70.65)
38.15
0.18
98.34
(32.90)
99.20 (0.90)
90.47 (13.85
1.36:32.50
19.51
0.50
BE
74.92 (49 29:
92.61 (7:37)
45.89 (2.94)
25.08 (49.28)
27:22
0.90
91.61
(25.67)
98.26 (3, 74)
85.53 (6.75)
14.96
1.36
72:40
(7:57)
2.500
27.40:446.00
76.02
0.83
05.05
(2.21)
17:39 (16.67)
fy-eann:
96.33 (70 70:
(4.52)
(6.36)
3.67 (70.60)
38.07
(16.64)
05.44 (4.55)
(4.33)
17.42
1.71
SiKoon
(3,82)
12.988
68.60)
(3.82)
(3.15)
1,00
Salun
(3,95)
(0.50)
(2.73)
2,45
0.46
(7.00)
35.
Fal
(000)
59.95 (0,00)
39.11
(3.00
0.35
05.64 (0.30)
99.99 (0.00)
78.01 (066))
33.36
Table I: Results of comparative results with two setups: training VGG-16 on CIFAR-10 with 10% forgetting rate and y 1: training VGG-16 on Tiny-ImageNet with 40% % forgetting rate and y 1/1/ (-) indicates performance gaps between approximate unlearning methods and the Retrain method across metrics. "std" is the variance of Avg. Gap across multiple experiments.
incorporating this balance factor, our final reweighting scheme can simultaneously perceive and mitigate both Heterogeneous & Skewed Unlearning Deviations.
Conclusion. Our method, FaLW, substantially mitigates the two types of unlearning deviation
identified in Observation 2 For the Meterogeneous Unlearning Deviation, the forgetting-awarr weight promotes the forgetting of under-forgotten samples by increasing their loss, while preventing further forgetting of over-forgotten samples by reducing their loss. For the Skewed Unlearning Deviation, the balance factor adjusts the sensitivity of this process, applying a more aggressive deviation correction for tail-class samples.
EXPERIMENTS
6.1 EXPERIMENTS SETTINGS
Datasets, Models, and Settings. Our empirical evaluation is conducted on image classifica- tion unlearning tasks. We utilize ResNet-18 (He et al.) 2016) and VGG-16 (Simonyan & Zisser- man (2014) as the primary model architectures. The experiments are performed on several standard benchmarks, including CIFAR-100 (Krizhevsky et al. 2009). CIFAR-10 (Krizhevsky et al., 2009). (Le & Yang 2015). We vary the size of the forget set from 10% to 50% of the total training data. To simulate imbalanced forget requests,
and Tiny-ImageNe
construct long-tailed forget sets where the class distribution follows a power law N& x km?, where N₁ is the number of samples in class k: We explore wide range of imbalance factors by setting y (0, 1/4, 1/2, 1, 1,3/2, where y = 0 corresponds to the balanced case(random sampling closer to).
Baselines and Evaluation. We compare our method against 9 baselines, including four gradient-
based methods:
Fine-Tuning (FT) (Warnecke et al., 2021). Gradient Ascent (GA) (Thudi et al.)
2022ab, Random Label (RL) (Golatkar al., 2020). SalUn (Fan al. 2024) and SFRon (Huang
methods (Chen et al. 2023): Boundary Expanding (BE) and Boundary Shrinking (BS); one influence-funcee
ct al. 2024); two boundary-basse
method: Influence Unlearning (IU) Izzo
ct al. 202T); and one parameter- sparsstctssaassiiiaattonnn method: LI-Sparse (Jia et al.) (202)). The detailed
algorithmic implementation of our proposed FaLW, as employed in the experiments, provided in Appendix C.1 To comprehensively evaluate performance, we use five primary metrics: O Forget- ting Accuracy (FA), the accuracy the unlearned model 0,, on the forget set D,; @ Membership Inference Attack (MIA), a privacy measure evaluated Dy to assess how effectively the model has forgotten the data; @ Retain Accuracy (RA), the accuracy on the retain set D,., which measures the model's 's fidelity the remaining data; O Test Accuracy (TA), the accuracy the original test set, reflecting the model's 's overall generalization ability; Avg. Gap, the average of the above four metrics.
8

---

Published as a conference paper at ICLR 2026
6.2 EXPERIMENTS RESULTS
Comparative Results. We conducted extensive experiments across multiple datasets, forget ratios, and imbalance settings. presentsssnnts the performance of FaLW against baselines, with two setups: VGG-16 trained on CIFAR-IO (10% forgetting rate, 7 = 1) and VGG-16 trained on Tiny- ImageNet (40% forgetting rate, 7 = 1/2). As shown, FaLW achieves a significantly lower average performance gap (Avg. Gap) compared to all other approximate unlearning baselines. Furthermore, FaL.W demonstrates superior performance across nearly all individual metrics (FA, RA, TA, and MIA). Additional results on Tiny-ImageNe and CIFAR-10, as well as under different forget ratios, different imbalance settings, and different model architectures, are available in Appendix C.2
Results for Various Degrees of Imbalance. To analyze the robustness under different imbal- ance scenarios, we conducted ex- periments CIFAR-100 with 30% forget ratio while varying the imbalance factor 7 from 0 (bal- anced distribution) to 2 (highly long-tailed). The results are sum- marized in T Across all de- grees of imbalance, FaLW consis- tently achieves lower Avg. Gap than SalUn, indicating that its per- formance more closely approxi- mates the Retrain gold standard. Furthermore, we observe telling trend in SalUn's 's behavior: at low imbalance levels, its Forgetting Accuracy (FA) is lower than Re- train's, s. suggesting over-forgetting As the imbalance becomes more severe, SalUn's 's FA surpasses that of Retrain, indicating shift to- wards under-forgetting. In con- trast, FaL.W maintains FAs much closer to the Retrain baseline re- gardless of the imbalance settings, This demonstrates that FaLW ef- fectively mitigates the heteroge- neous unlearning deviation that plagues existing methods.
Method
FA
RA
TA
MIA
Avg. Gap
Retrain
59.11
99.95
59.74
40.89
0.00
7 = 0
SaLUn
56.93
99.96
60.60
44.06
1.55
FaLW
59.09
99.98
61.38
41.91
0.68
Retrain
57.13
99.96
59.24
42.87
0.00
7=
SaLUn
55.19
99.96
59.98
39.41
1.54
FaLW
57.73
99.98
60.67
41.27
0.91
Retrain
56.29
99.97
59.33
43.71
0.00
SaLUn
$4.00
99.98
59.65
40.18
1.54
FaLW
56.13
99.98
61.77
42.87
0.86
Retrain
50.36
99.95
57.71
49.64
0.00
==
SaLUn
48.47
99.96
58.66
51.53
1.19
FaLW
50.76
99.98
58.62
47.24
0.93
Retrain
19.30
99.96
50.70
80.70
0.00
7
SaLUn
21.75
99.97
53.95
78.25
2.04
FaLW
19.69
99.97
52.63
79.31
0.93
Retrain
4.43
99.95
48.13
95.57
0.00
SaLUn
5.75
99.96
51.00
96.25
1.22
FaLW
4.64
99.87
50.47
96.36
0.85
Retrain
1.82
99.94
47.12
98.17
0.00
==2 2 SalUUn
4.79
99.96
50.32
95.20
2.29
FaLW
2.96
99.85
49.95
97.04
1.30
Table 2: Unlearning performance metrics of SalUn and FaLW with ResNet-18 architecture trained on CIFAR-100 by 30 % unlearning ratio, under varying imbalance levels of the forget set controlled by 7 (larger 7 denotes greater class imbalance in the forget set).
Ablation Study on Balance Factor. We conduct an ablation study to analyze the ben- efits of Balance Factor. The experiments are performed on CIFAR-100 with ResNet-18 and a 30% forget ratio, focusing on two highly skewed scenarios with IFs ? 1.5 and y 2.0. In Table 3 the findings reveal crucial trade-off: while the inclusion the balancing factor can cause minor decline in the forget- ting accuracy for head-class data, leads to substantial improvement for tail-class data. This result confirms that the Balance Factor effectively fulfills its designed purpose to specifically target and alleviate the skewed unlearning deviation.
7
Bolance Factor
1.5
x
0.18
0.51
.9.98
-12.19
0.69
-8.04
X
1.08
1.22
.12.56
-18.76
2
L.4
1.35
-10.71
-13.04
Table 3: Forgetting Accuracy gaps (AFA) vs. Re- train on CIFAR-100 (30% unlearning, ResNet- 18) with a power-law forget set (N& x 1.5 and Ng. x k -22. Gaps are shown for the overall set (AFA) and head/mid/tail classes.
Analysis of FaLW's Effectiveness.. Figure P validates FaLW'ss effectiveness in addressing un- learning deviation issues. Regarding Heterogeneous Unlearning Deviation, FaLW consistently

---

Published as a conference paper at ICLR 2026
duces the performance gap whether the baseline over-forgets under-forgets, correcting deviations in both directions. For Skewed Unlearning Deviation, it successfully mitigates the severe over- forgetting of tail-class samples that plagues the baseline in highly long-tailed settings.
Further Experiments. We provide additional comparative experiments in Appendix C.2 t-SNE
visualizations in Appendix C.3 plug-and-play characteristic analysis in Appendix F rameter sensitivity analysis for r in Appendix G and an analysis on the choice normal distribu- tion in Appendix H
hyperpa-
CONCLUSION
To address Heterogeneous and Skewed Unlearning Deviation in long-tailed unlearning, we propose FaLW, plug-and-play dynamic loss reweighting that adaptively adjusts the unlearning intensity for each instance. Extensive experiments demonstrate that FaLW achieves SOTA performance.
8 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of work, providing the code for experiments part of the supplementary material. We also provide a description of relevant implementation details in Appendix C.T
ETHICS STATEMENT
Our work adheres to the ICLR Code Ethics. This research did not involve human subjects or animal experimentation. All datasets used in this study are publicly available and were handled in strict accordance with their usage terms to protect privacy. We have taken proactive measures mitigate potential biases our methodology and evaluation, ensuring no personally identifiable information was processed and no discriminatory outcomes were generated. We are committed to maintaining transparency and integrity throughout our research process.
10 ACKNOWLEDGEMENTS
The authors gratefully acknowledge the support from the National Natural Science Foundation of China (NSFC) under Grant Nos. 62402472, and 12227901. This work was also supported by the Natural Science Foundation of Jiangsu Province (No. BK20240461), the Project of Stable Support Youth Team in Basic Research Field, CAS (No. YSBR-005), and the Academic Leaders Cultiva- tion Program at USTC. The Al-driven experiments, simulations and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
REFERENCES
Keshav Balasubramanian. Abdulla Alshabanah, Elan Markowitz, Greg Ver Steeg, and Murali An- navaram. Biased user history synthesis for personalized long-tail item recommendation. Pro- ceedings of the 18th ACM Conference Recommender Systems, pp. 189-199, 2024.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1138-1156. IEEE, 2022.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pp. 141-159. IEEE, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution aware margin loss. Advances neural information processing systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2026
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song. Andreas Terzis, and Florian Tramer. Mem- bership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pp. 1897-1914, IEEE, 2022.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7766-7775, 2023.
Xinwen Cheng, Zhehao Huang, Wenxin Zhou, Zhengbao He, Ruikai Yang, Yingwen Wu, and Xi- aolin Huang. camnaininn--ata-fieee machine unlearning by suppressing sample contribution. arXiv preprint rrXiv:2402.1510% 2024.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7210-7217, 2023.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. on effective number samples. In Proceedings of the IÉEE/CVF conference on computer vision and patlern recognition, pp. 9268-9277, 2019.
loss based
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation. International Conference on Learning Representations, 2024.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC confer- ence on computer and communications security, pp. 1322-1333. 2015.
Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. A swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 11 1139-1147. PMLR, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9304-9312, 2020.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.0300 2019.
Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling ap- proach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pp. 1322-1328. leee, 2008
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yin- He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 235-244, 2021.
Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what it is and what it means. Information Communications Technology Law, 28(1):65-955 2019.
Hongsheng Hu, Zoran Salcie, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Mem- bership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54 (11s):1-37, 2022.
11

---

Published as a conference paper at ICLR 2026
Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient-based machine unlearning with remain geometry enhancement. Ad- vances in Neural Information Processing Systems, 37:26377-26414 2024.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International conference on artificial intelligence and statistics, pp. 2016-2016. PMLR, 2021.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584-5148 51605, 2023.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafilluu. Towards unbounded machine unlearning. Advances neural information processing systems, 36: 1657-97-1987, 2023.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7,7):3, 2015.
Zheng Li and Yang Zhang. Membership leakage in label-only exposures. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 880-895, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2537-2546, 2019.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Seth Neel, Aaron Roth, and Saced for machine unlearning. In Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
Descent-to-delete: Gradient-based methods
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and
forgetting functions. Psychological review, 97(2):285, 1990.
Protection Regulation. General data protection regulation. Intouch, 25:1-5, 2018.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Pro- cessing Systems,
2021.
Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape machine unlearning: A comprehensive survey and taxonomy. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp. 241-257, 2019.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effec- tive machine unlearning. IFEE Trunsactions on Neural Networks and Learning Systems, 35(9): 13046-13055. 2023.
Anvith Thudi, Gabriel Deza, Varum Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Un- derstanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (FuroS& P), pp. 303-319. IEEE, 2022a.
12

---

Published as a conference paper at ICLR 2026
Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX security symposium (USENIX Security 22), pp. 4007-4022 2022b.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, (668):5362-5388) 2024.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprini vrXiv:2108.1157, 2021.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self- knowl- edge distillation for long-tailed classification. In European conference on computer vision, pp- 247-263. Springer, 2020.
Jie Xu. Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 8(3):155021168, 2024.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learn- ing for face recognition with under-represented data. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 5704-5713, 2019.
Kairan Zhao, Meghdad Kurmanji, George-Octavian Octavian Bärbulescu, Eleni Triantafillou, and Peter Tri- antafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37:12293-12333, 2024.
13

---

Published as a conference paper at ICLR 2026
Appendix
FaLW: A Forgetting-ane -ware Loss Reweighting for Long-tailed Unlearning
This appendix organized as follows:
Section A Proofs. This section provides the theoretical proofs for Proposition and Proposi-
tion
2
Section Observations. This section presents additional experimental results for Observa- tion and Observation to demonstrate the generality of our findings.
Section Experiments Complement. This section details supplementary experiments, in- cluding Implementation Details, Additional Comparative Results, and a t-SNE Visualization Analysis.
Section D: Limitations. This section discusses the limitations of the proposed FaLW method and outhines potential directions for future work.
Section LLM Usage Statement. This section provides statement on the use of Large Language Models (LLMs) in this research.
Section F Plug-and-Play Characteristic Analysis of FaLW. This section provides an analysis the plug-and-play nature of the FaLW method, demonstrating its effectiveness when integrated with existing unlearning baselines.
Section G Hyperparameter Analysis for т. This section conducts a sensitivity analysis for the hyperparameter to evaluate its impact on the overall unlearning performance.
Section H Clarification about use of the normal distribution. This section justifies the choice of the Normal distribution for modeling predictive probabilities and provides a comparative ex- periment with bounded Beta distribution variant.
A PROOF
A.1 PROOF FOR PROPOSITION I
Proposition 1 (Proposed in paper). When an approximate unlearning method has access only to the forget set Dj, it is prone to severe over-forgetting. Formally, for an unlearned model O.. M(,,, Dj), , the condition for over-forgetting. Po., (c(t;) < <poo (c|x.) (oc(x.))+0 - TT, is frequently met for samples (Ii, c) E Dy.
Proof. When only D, is available, approximate unlearning methods primarily update the model by applying gradients derived from this set, often designed to reverse the original training process. We illustrate this using the Random Labeling (RL) method as example. In RL, for any sample (I;, y/s = <) E Dy, its label is changed to an incorrect class c' * c. The model then performs gradient descent loss function computed with this incorrect label, which approximates applying an "unlearning" gradient. The standard cross-entropy loss for this relabeled sample is:
CRI(2) = - log(Per)
where RC are the logit outputs of the model for sample 24, and pj e² Ee Ek= e is the softmax probability for class j. The gradients of this loss with respect to the logits of the true class and the incorrect class c' are: HLRL
[Pe>
if if j=
0zj
Pe 1,
From Eq. 9 we can see that the unlearning process, driven by gradient descent on CRL, continuously pushes the probability of the true class, Pe, towards zero. The magnitude this gradient, Pe, is larger when the model is more confident, accelerating the process. This dynamic, however, is misguided. Forgetting a sample is not equivalent to demanding that the model's confidence on it drops to zero. A model that has faithfully forgotten क; should still retain knowledge of class from the (now absent)
14

---

Published as a conference paper at ICLR 2026
retain set, granting it general capability recognize class Even after unlearning, the model expected to have some non-zero predictive capability for a; based on this general knowledge. Therefore, when only D is available, the unlearning process lacks a proper stopping condition and prone to over-forgetting. This conclusion holds for other gradient-based methods as well, as they differ mainly in how the "unlearning" gradient is computed, not in the gradient update process. L
A.2 PROOF FOR PROPOSITION 2
Proposition 2 (Proposed in paper). All else being equal, a model trained on dataset including sample (a Ys = c) will exhibit higher confidence on that sample's 's true class compared to a model trained on the same dataset excluding it. Formally, let O, be the parameters of model trained on D, and let 0₂ be the parameters of model trained on DD (()) Then, it generally holds that:
po. po. (c(24) Z Po. (c/x.)
(10)
Proof. The proof is based the principle of minimizing loss function during model training. We use the standard cross-entropy loss (or negative log-likelihood) for classification. The cross-entropy loss for single sample (zj, Mj) given by:
{(0; zi, y1) - log pe ( (x;)
Let D. D ((x, yi = c)). The parameters 0_ for the model trained on D_ are the solution to the following optimization problem:
arg c.) where (0)
Σ
L(0:2z,8;)
(z,,),)DD.
The parameters 0, for the model trained on the full dataset D = D, U (f ys = c)} are the solution to:
0. arg min (0)
where the total loss (0) can be expressed as:
(.) = L. (0) L(0: Fz. 34 =
By the definition of 0, and 0, as the minimizers of their respective loss functions, have the following two conditions:
1. Since O, minimizes (0), its value 0, must be less than equal to its value any other point, including 0,:
(0.) s (0.)
(11)
2. Similarly, since 0, minimizes (0):
L. < (0.)
(12)
We begin with the first optimality condition equation 11 and substitute the definition of Co: (..)) + L(O.;x;;,)) < (O.)) + L(0.;
Rearranging the terms, get:
L(0,; x4, I(O,;a,,e) S (..) (0.)
From the second optimality condition equation 12 we know that the right-hand side, L. (..) L. (0.), is less than or equal to zero. Therefore:
I(0,; I,; c) T((0;; 3i, c) $
Substituting the definition of the loss function L:
log (c|æ:) S - log Po. (c/xz)
Since the logarithm function is monotonically increasing, this inequality holds if and only if:
Pe. (c|2z) 2 Po. (c|2.)
15

---

Published as a conference paper at ICLR 2026
Cifar-100
Cifar-100
20% Forget Ratio
30% Forget Ratio
10
10
5
Density
Density
0
5
10
SalUn
Retrain
15
SalUn
Retrain
20
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prediction Probability
Prediction Probability
Tiny-Imagenet 20% Forget Ratio
Tiny-Imagenet 50% Forget Ratio
20
20
10
Density
Density
0
10
10
20
Retrain
SalUn
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prediction Probability
Prediction Probability
Figure 4: Comparative results of predicted probability distributions on randomly sampled forgotten data between the Salun and Retrain methods under multiple settings.
B OBSERVATION
B.1 EXPERIMENTS FOR OBSERVATION
1
Observation (Proposed paper). The predictive probabilities of the unlearned model O. pro- duced by SalUn are consistently lower than those of the retrained model 0, across various experi- mental settings. This indicates that despite leveraging the retain set for capacity preservation, the SalUn method still exhibits discernible degree of over-forgetting.
Further results, presented in Figure 4 provide additional evidence of baseline limitations. It is observed that across multiple settings, the predictive probabilities of SalUn on the forget are consistently and significantly lower than those of the Retrain. This demonstrates that even when leveraging the retain set and applying parameter constraints, the SalUn method is still prone to severe over-forgetting, Overall, the experimental results are consistent with our proposed Observation I
Head
Retrain
Retrain
Aec Gap
Skewed
OOve!
Uleday
Retrain
65%
Anart
Figure 5: Performance discrepancies between the SalUn method and Retrain the head, middle, and tail segments of forgotten data under different long-tailed configurations across various experi- mental settings.
B.2 EXPERIMENTS FOR OBSERVATION
2
Observation 2 (Proposed in paper). Under a long-taaies forget distribution, approximate unlear- ing methods exhibit two key essences:
16

---

Published as a conference paper at ICLR 2026
Heterogeneous Unlearning Deviation: The model demonstrates disparate unlearning pat- terss across classes, rypically under-forgetting samples from head classes while over- forgetting those from tail classes.
The magnitude of the unlearning deviation is disproportion- ately larger for tail-class data compared to head- and medium-clad classs data.
Skewed Unlearning Deviation:
We conducted further experiments under various imbalance levels and other settings, with detailed results shown in Figure A consistent pattern emerges: the degree of imbalance increases, the unlearning behavior for head-class data generally shifts from over-forgetting to under-forgetting, corroborating our first point in Observation 2 Conversely, for tail-class data, the unlearning per- formance deteriorates further as the imbalance becomes more pronounced, leading to more severe -err-forget This confirms our second point in Observation Thus, Observation 2 is in line with the results of large number of experiments.
Algorithm The FaLW Algorithm (RL-based version)
Require: Original model parameters O, Forget set Dy; Retain set D,; Validation set Deat- Require: Hyperparameters: learning rate T/rr, exponents 7, 7, number of epochs E. с 0.
Construct relabeled forget set D' - {I (z) w/) E Dy. + yi
Construct training set for unlearning D Pantearn f U.
Pre-compute class-wise balancing factor Bg for each class c using Eq. (6) for epoch + 1... E' do
for each batch Dunlearn do
Estimate target distribution parameters Hec, T. for each class from predictions on Doal Partition the batch into forget part by and retain part b,
Calculate retain loss {Lr,i) +- loss(b,; 0)
Calculate forget loss {C and probabilities {rs} on by Calculate weights {wya) for samples in by using ply de {p.h. B. via Eq. (7)
Liotal f mean(fw,, C,,, {Cr+))
Combine losses
O 0- 0 Tre V Ctoial
Update parameters via SGD
end for
end for return 0
C EXPERIMENTS COMPLEMENT
C.I IMPLEMENTATION DETAILS
For all methods, we use an SGD optimizer with a momentum of 0.9, weight decay of 5e-4, and batch size of 512. For the Retrain, we train for 150 epochs with an initial learning rate of 0.01, which decayed by a factor of 10 epochs 90 and 120. For all baselines. we unlearn for 10 to 20 epochs, and the learning rate is tuned within the range of [0.0005, 0.02], with generally lower and higher rates for CIFAR-10 and CIFAR-100. For SalUn (Fan al.)
rates used for 2024), use the official implementation's 's default setting, which prunes the top 50% of parameters based on gradient magnitude. Similarly, for SFRon (Huang et al. 2024). we reproduced our results following the official code and hyperparameter settings provided in its original publication. The hyperparameter 5 for our balancing factor in FaLW is tuned within the range 10.1, 0.2]. The pseudo- code for FaLW is provided in Algorithm and our source code is included in the supplementary materials.
C.2 MORE COMPARATIVE RESULTS.
We conducted a comprehensive evaluation FaLW against baselines on CIFAR-100, CIFAR-10, and ImageNe,,, performing multi-faceted comparison across various forget ratios, imbalance levels, and model architectures. Table 4 presents the results for the ResNet-18 architecture on CIFAR-10M 100 across multiple settings. The results for the VGG-16 architecture CIFAR-10 and
17

---

Published as a conference paper at ICLR 2026
Iny-ImageNet are detailed in T and Table respectively. The findings consistently show that our method, FaLW, outperforms the baselines in all evaluated settings, reducing the Avg. Gap by up to 3%.
Retrain
FalW(Ours)
SalUn
60
60-
40-
40
40-
Dimension
Dimension 2
Dimension
20-
0-
-40
40
400
-60
Retain Data
-60
Recain Data Forpet Data
-60-
Rebain Data
Forget
Data
-60 400
-00
20
40
60
80
-60
-000-40-00 -0
20
40
60
80
-60
40
-00
0
20
40
60
80
Dimension
Dimension
Dimension
Figure 6: Experimental setup: 20% of the Cifar100 dataset is randomly selected the forgetting set, following a data distribution of W₁. x 0.25 Under this setting, we select 2 head classes, middle classes, and tail classes, and visualize their representations via t-SNE across three methods: Retrain, SalUn, and FaLWW Each 'O' represents sample in Dr, while each x denote sample from D-
C.3 T-SNE VISUALIZATION ANALYSIS
To analyze the characteristics of forgotten data in the embedding space after approximate unlearning, we use 1-SNE visualize the representation. The experiment is set with ResNet-18 trained on CIFAR-100 by 20% forget set following a distribution of Nr. k-0.25 We visualize samples from two heads, two medians, and two tails, with the results presented Figure 6 is evident that the feature distribution of FaLW closely resembles that of the gold-standard Retrain model. Specifically, for the forgotten samples, FaLW, much like Retrain, does not aggressively push their representations away from their original class clusters, thus achieving faithful unlearning. In stark contrast, SalUn displaces the forgotten samples far from the Retrain data clusters, exhibiting severe over-forge
D LIMITATIONS
While our proposed FaLW method demonstrates notable success in addressing and correcting devi- ations machine unlearning, we acknowledge primary limitation: its reliance on auxiliary data to evaluate the model's s state during the forgetting process. This dependency external data may limit the applicability FaLW in resource-constrained or data-sensitive environments. Addressing this limitation constitutes crucial direction for future work.
E LLM USAGE STATEMENT
In the preparation of this manuscript, we utilized large language model (LLM) as general-purpont writing assistant. The role of the LLM was strietly limited improving the language and readability of our paper. This included tasks such as correcting grammatical errors, refining sentence structure for clarity, and enhancing overall prose. We confirm that the LLM did not contribute the core research ideas, experimental design, data analysis, or the generation of any substantive content. All intellectual contributions, including the concepts, methodology, and conclusions presented this paper, are solely the work of the human authors.
F PLUG-AND-PLAY CHARACTERISTIC ANALYSIS OF FALW
As mentioned the main text, FaLW is designed as plug-and- -nd--lay method. is not a standalone unlearning algorithm but rather a forgetting-aware loss reweighting strategy intended for integration
18

---

Published as a conference paper at ICLR 2026
with existing methods. To validate the efficacy of FaLW plug-and-pln universal module, integrated it into four baseline methods: GA (Thudi et al. 2022a). RL GGolatkrr et al. 2020). SalUn (Fan et al. 2024), and SFRon (Huang et al., 2024). In terms of implementation, since SFRon also proposed a weighting term, we replaced its original dynamic weighting method with our Wa dynamic weighting. For the other three methods, we directly applied our weighting the loss of the forget set samples. We conducted experiments on the CIFAR-I00 dataset with the ResNet- 18 architecture, 20% forget rate, and 7 1/4. The performance comparison before and after integrating FaLW is presented in Table 7 As the results show, the introduction of FaLW brings unlearning performance improvements all baseline methods.
100.00
64
Reiain
92.5
37.5
92.0
37.0
12-
10.25
61.
96.6
61.0
36.0
0.00.10 15..110.55.000 0.20 05.25.000
100
10010000115. 25 10
20 0F
0020:10016020000
Figure 7: Hyperparameter sensitivity analysis for CIFAR-100 (ResNet-18, 7 1/4, 20% forget rate). The dashed red line represents the "Retrain" baseline performance for each metric.
G HYPERPARAMETER ANALYSIS FOR
We conduct a sensitivity analysis for the hyperparameter т, which controls the exponent of the balance factor. The analysis is performed on the CIFAR- 100 dataset using a ResNet- 18 model, with 20% forget rate and an imbalance ratio of 1/4. We vary within the range [0. 1.0]. The results are illustrated in Figure 7 The performance across all metrics (FA, RA, TA, and MIA) is strongest and most stable when the model's deviation correction becomes overly sensitive, leading to a degradation in unlearning performance.
within the 10. 0.2] range. As increases beyond this optimal range,
H CLARIFICATION ABOUT USE OF THE NORMAL DISTRIBUTION
Our initial choice of the Normal distribution was driven by practicality and robustness. We employ it not to perfectly fit the data density, but to derive stable, efficient metric for "deviation" (z-score). In dynamic optimization scenarios, the Gaussian assumption offers computational simplicity and stable parameter estimation (μ4 σ), which helps avoid gradient instability.
To evaluate the impact of using a distribution with matching support of [0, 1], we implemented variant named FaLW-beta. In this variant, we replace the Gaussian model with Beta distribution. We estimate the parameters a and 3, via Maximum Likelihood Estimation (MLE) on the validation set and adapt our dynamic weight formulation using the Beta Cumulative Distribution Function (CDF), F(p)) = BetaCDF(ps (ake; Bz). We then map the probability to a deviation score 8; E [-1,
Wi + sign(s:) ((s:)) where 8; (F 0.5)
(13)
We compared the FaLW-normal and FaLW-beta variants the CIFAR-100 and Tmay-Im datasets. The results are summarized in Table While FaLW-beta yields reasonable performance, it consistently underperforms FaLW-normal, exhibiting higher Average Gap. Empirically, observed that the flexibility of the Beta distribution leads to high sensitivity and erratic weight fluc- tuations during the unlearning process, likely due to estimation errors ar and 3c. In contrast, the Normal distribution acts as a robust proxy, providing smoother gradients and greater stability "out of the box." " Consequently, we retain the Normal distribution in our primary method for robustness.
19

---

Published as a conference paper at ICLR 2026
Table 4: Results on the CIFAR-100 dataset, with the forget set following a distribution of Ne: o
k4225 (-) indicates performance gaps between approximate unlearning methods and the Retrain method across metrics.
CIFAR-100, ResNet-18
1/4
Method
Random Forget (10%)
MIA
Avg. Gap
Retraän
64.82 (0.00)
99.95 (0.00)
64.24 (0.00)
35.18 (0.00)
0.00
RL.
31.077 (33.76)
91.52 (2.42)
62.22 (2.022
48.93 (13.76)
12.99
GA
95.22 (30.40)
96.66 (3.29)
68.83 (4.59)
4,78 (30.40)
17.17
BE
96.71 (31 89)
96.81 (3.13)
69.61 (5.37)
3.29 (31.8))
18.07
BS
96.53 (31.71)
96.60 (3.35)
68.88 (4.64)
3.47 (31.71)
17.85
ly-spanse
(0.
SPRon
62. (2.13)
99.98 (0.03)
66.70 (2.46)
38.31 (3.13)
1.94
SalUn
(0,500
99.95 (0.00)
63.39 (0.85)
32,20 (2,98)
1,20
Fail
(0.30)
99.98 (0.03)
86 (1.62)
33.58 (.60)
CIFAR-100 100. ResNet-18
1/4
Method
Random Forget (20%)
Avg. Gap
Retrain
62.09 (0.00)
99.95 (0.00)
62.10 (0.00)
37.91 (0.00)
0.00
FT
96.61 (34.52)
99.93 (0.02)
69.98 (7
3.39 (34.52)
19.24
RL
47.49
96.99 (2.96)
(0.64)
(4.60)
5.70
GA
98.22 (35.13)
96.66 (3.30)
68.83 (6.75)
4.78 (33.13)
19:07
IU
97.02 (34.93)
96.91 (3.05)
70.57 (8.47)
2,98 (34.93)
20.35
(33.32)
BS
87.04 (24.95)
37.67 (12.29)
58.13 (3:97)
12.96 (21.96)
16.54
l--sparse
94.02 (31.93)
93.88 (6.07)
65.51 (3.41)
5.98 (31.93)
18.34
SHRon
(2.38)
96.92 (3.03)
(1.99)
(2.38)
FaiW
61.74 (0.34)
99.98 (0.02)
62.69 (0.59)
38.26 (0.34)
0.33
CIFAR-100 ResNet-11.
1/4
Method
Random Forget
FA
RA
TA
MIA
Avg. Gap
Retrain
57.13 (0.00)
99.96 (0.00)
59.34 (0.00)
42.87 (0.00)
0.00
96.32. (39.199)
99.95 (0:01)
(10.56)
3,68 19)
22.23
RI.
24.24 (12.99)
96.34 (3.62)
58.08 (8.08(1.16)
65.76 (22.6))22))))
10.14
GA
K5.51 (28.38)
89.24 (10.72)
61.40 (2.00)..))) 16)
14.49 (28
17.41
97,02
75.60 (18.47)
79.53 (20.13)
18.09 (11.15)
24.10 (18.47)
17.13
BS
$6.79 (0.34)
58.94 (41.01)
38.46 (20.78)
43.21 (0.34)
15.62
fu-sparse
(38.50)
95.38 (4.58)
66.65 (7.44)
4,36 (38.50)
22.25
SalUin
55.19 (1.95)
99.96 (0.00)
59.98 (0.74)
39.41 (3.46)
1.54
FaiW
57.73 (0.60)
99.98 (0.02)
60.67 (1.43)
41.27 (1 (1.60)
0.91
CIFAR- 100, ResNet-18.
Forget (40%)
FA
RA
TA
MIA
Avg. Gap
Rernin
53.01 (0.00)
99.95
56.41 (0.00)
(0.00)
0.00
FT
96.24 (43.23)
99.96 (0.01)
69:46 (13.05)
3,76 (43.23)
24.88
RL
33.86 (19.15)
95.17 (4.78)
54.51 (1.90)
619.1
11.25
96.88
IU
96.96 (13.95)
96.91 (3.04)
70.27 (13.86)
3.04 (13.95)
26.20
BE
96.92 (43.91)
96.85 (3.10)
(13.99)
3.0M (43.91)
26.21
BS
96.92 (43.91)
96.88 (3.07)
70.23 (13.82)
3.08 (43.91)
26.18
L--spare
SFRon
$1.33 (1.68)
99.98 (0.03)
58.59 (2.18)
48.67 (1.68)
1.30
SalUn
$1.27 (1.74)
99.97 (0.02)
$7.89 (1 A8)
49.73 (2.74)
1.49
53.32(0.31)
(0.03)
57.71 (1.30)
(1.31)
ResNet-11.
1/4
Method
Random Forgel (50%)
FA
RA
TA
MIA
Avg.
Retrain
48.44 (0.00)
(9.00(
51.46 (0.00)
51.56 (0.00)
0.00
FT
95.79 (47 35)
99.96 (0.02)
69. 19.15 69)
5.78 (45. 78)
27.71
67.78
(23.15)
GA
96.56 (48. 12)
96.86 (3.08)
69.10 (17.64)
3.44 (48.12)
29.24
I0
96.98 (48.54)
96.56 (3.07)
70.26 (18.80)
3.02 (48.54)
29.74
BE
96.90
96.93 (3.01)
70.11 (18.65)
3.07
29.65
96.92.
96 96.88 (3.06)
(18.84)
3,08 (48.48)
29.71
4-sparse
75:32 (26 88)
88.76 (11.18)
61. (10.08)
24.68 (26.89)
14.76
SFR
16.30 (2.14)
99.98 (0.01)
54.78 (3.32)
$3.70 (2.144
1.91
SalUn
FaiW
48.06 (0.38)
99 (0.04)
$2.05 (0.59)
(1.38)
0.601
20

---

Published as a conference paper at ICLR 2026
Table 5: Results the Tiny-Imagenet dataset. (-) indicates performance gaps between approximate unlearning methods and the Retrain method across metrics.
Tay- Imugenet. Vgg- y
Method
Random Forget (20%)
Avg. Gap
Retrain
48.87 (0.00)
99.98 (0.00)
48.51 (0.00)
51.13 (0.00)
0.00
FT
95.62 (46.76)
99.98 (0.00)
55.27 (6.76)
4.38 (46.75)
25.07
RL
32.47 (16-10)
98.25 (1.73)
45.37 (3.14)
67.53 (16.40)
9.42
GA
94.61 (45.74)
95.41 (4.57)
54.27 (5.76)
5.39 (45.73)
25.45
IU
98.64 (46.77)
95.56 (4.43)
55.55 (7.04)
4.36 (46-77)
26.25
BE
94.41 (45.54)
94,27 (5.71)
49.87 (1.36)
5.59 (45
24.54
BS
92.26 (43.39)
92.45 (7.54)
48.25 (0.26)
7.74 (43.39)
23.64
f-squrse
95.68 (46.82)
95.53 (4.45)
55.47
4.32
26.26
STRon
50.23 (1.33
99.98 (0.00)
50.11 (1.60)
50.77 (0.36)
0.83
SalUn
(6.13)
99.13 (0.85)
46.59 59 (1.92)
56.26 (5.16)5.33)
3.51
(8.27)
(0.00)
(2.16)
(0.70
Tiny Imagenett Vgg-16.
Method
Raudom Forget (50%)
RA
TA
MIA
Gap
Retrain
40.60 (0.00)
99.99 (0.00)
40.63 (0.00)
59.40 (0.00)
0.00
FT
95.59 (54.99)
99 99.99 (0.00)
54.71 (14.08)
4,41 (54.99)
31,02
96.91
(3.30)
GA
95.58 (54.97)
95.49 (4.51)
55.45 (14.82)
4.42 (54,97)
32.32
95.65 (55.05)
95.50 (4.50)
55.65 (15.02)
4.34 (55.05)
32.41
(55.04)
(4.52)
(14.66)
(55.04)
BS
95.66 (55.05)
95.47 (4.53)
55.03 (14.40)
4.34 (55.05)
32.26
1--ppase
95.68 (58.08)
95.48 (4.52)
58.45 (14.22)
4.32 (55.08)
32.37
SFRoa
38.89 (1.71)
99 99.99 (0.00)
40:45 (0.18)
(1.70)
0.90
SalUn
38.49 38.19 (2.11)
97.21 (2.78)
36.23 (4.40)
61,51 (2.1
2.85
FaiW
(1.16)
(0.00)
40.53
(2.16)
Tiny- Imagenet, Vgg-16. .
1/3
Method
Random Forget (30%)
EA
RA
TA
MIA
Gap
(0:00)
99.98 (0.00)
(0.00)
(0.00)
FT
90:44 (45.87)
99.74 (0.25)
$1.19 (5.58) (5.58)
9.56 (45.87)
24.39
RL
31.76 (12.81)
97.13 (2.85)
41.25 (4.36)
68.24 (12.81)
8.21
77.98 (33.41)
93.73 (6.26)
48.67 (3.96)
22.02 (31.41)
19:03
IU
95.85 (51.28)
95.46 (4.52)
55.67 (10.06)
4.15 (51.28)
29.29
94.89 (50.32)
94.94
$1.43 (5.82)
5.11 (50.32)
94.33 (49.76)
94.59 (5.39)
50.81 (5.20)
5.67 (49.76)
27.53
1y-sparee
95,83 (51.26)
95.46 (4.53)
55.17 (9.56)
4.17 (51.25)
29.15
SFRon
$1,20 (6.63)
99.79
45.49
(6.63)
3.39
SalUn
(4.69)
99.95 (0.03)
43.47 (2.14)
60.12 (4.69)
2.89
FaiW
46.94 (2.37)
99.98 (0.00)
46.11 (0.50)
$3.06 (2.37)
1.11
Tiny- Imagenet, Vgg-16
Method
Random Forget (10%)
RA
TA
MIA
Avg. Gap
Retrain
25.63 (0.00)
99.98 (0.00)
48.83 (0.00)
74,36 (0.00)
0.00
86.97 (61.33)
99.98 (0.00)
53.95 (5.
13.03 (61.33)
31.95
RL
18.50 (7.13)
99.83 (0.15)
48.51 (0.32)
81.50 (7.
3.69
GA
77.98 (52.34)
93.73 (6.26)
48.67 (0.
22.02 (S2.34)
27.77
IU
96.29 (70.66)
95.46 (4.52)
55.71 (6.88)
3.71 (70.65)
38.18
BE
74.92 (49.29)
92.61 (7.37)
45.89 (2.94)
25.08 (49 28)
27.22
1i-ppare
96.33 (70. 70)
95 (4.52)
$5.19 (1.9)6.33) (6.36)
3.67 (70.69)
38.07
SFRoa
22.01 (3.62)
97.10 (2.88)
44.23 (4.60)
77.98 (3.62)
3.68
SalUn
(2.95)
(0.80)
(2.72)
71.02 (3.34)
FalW
(0.07)
99 (0.00) 49.11 (0.28)
73.30 (1.06)
0.35
Table 6: Results the Cifar-10 dataset. (-) indicates performance gaps between approximate unlearning methods and the Retrain method across metries.
CIFAR-10, Vig- . 1/2
Method
Random Forget (40%)
FA
RA
TA
MIA
Avg. Gap
(0.00)
(0.00)
34.06 (0.00)
FT
68.42 (2.48)
96.51 (3.48)
78.37 (0.25)
31.58 (2.48)
2.18
RL
72.37 (6.43)
98.87 (1.13)
81.15 (2.53)
27.63 (6.43)
4.13
58.67 (7.27)
84.28 (15.71)
67.21 (11.41)
(1.23
10.42
IU
98.8 90))
(0.20(0..8)
90.47 (11.85)
(32500
19.61
BE
91.61 (25 67)
98.26 (1.6661.)) 74)
65. (6.76)
8.39 (25.67)
14.96
BS
82.61 (16.67)
95.93 (1.05)
80.83 (2.21)
17.39 (16.67)
9.90
h-sparse
82.58
(16.66))66)
SFRon
68.05 (2.14)
97.10
82.15 (3.53)
30.91 (3.15)
2.93
SallIn
63.84 (2.09)
96.04 (3.96)
77.03 (1.59)
33.16 (0.91)
2.14
FaiW
65.64
99.99 (0.00)
78.01 (0.61)
33.36 (0.70)
0.40
21

---

Published as a conference paper at ICLR 2026
Table 7: Plug-andPPlay Experimental Results for FaLW. Comparison of baseline methods before and after FaLW integration under CIFAR-I00, ResNet-18, ? 1/4, 20% Random Forget setting.
Method
FA
RA
TA
MIA
Avg.Gap
Retrain
62.09
99.95
62.10
37.91
0.00
GA
95.22
96.66
68.83
4.78
19.07
GA FaLW
89.678
90.625
63,48
10.32
16.47
RL
47.49
96.99
61.46
42.51
5.70
RL + FaLW
62.244
99.969
66.46
37.75
1.17
SFRon
65.30
99.98
66.42
33.69
2.95
SFRon FaLW
62.856
99.653
62.77
37.144
0.62
SalUn
59.71
96.92
60.11
40.29
2.44
SalUn FaLW
61.74
99.98
62.69
38.26
0.33
Table 8: Comparison between Fan-mnoomml and FaLW-heta variants on CIFAR-100 and Tiny- ImageNet.
Dataset (Setting)
Method
FA
RA
TA
MIA
Avg. Gap
CIFAR-100
Retrain
62.09
99.95
62.10
37.91
0.00
(ResNet- 18, 7 = 1/4)
Fal--norrml
61.74
99.98
62.69
38.26
0.33
(20% Forget)
FaLW-beta
59.96
99.95
63.06
39.04
1.06
Tiny-ImageNNt
Retrain
25.63
99.98
48.83
74.36
0.00
(VGG-16, 7
Fal--anormal
25.70
99.98
49.11
73.30
0.35
(10% Forget)
FaLW-beta
26.91
99.97
48.77
72.09
0.91
22


Reviews:
[{"review_id": "rYEycD6Zw4", "reviewer": "Reviewer_V5ik", "summary": "This paper tackles the problem of machine unlearning under long-tailed data distributions in the forget set. It demonstrates that in real-world scenarios, the data to be forgotten can be highly skewed. It identifies two interesting  phenomena while unlearning in this situation, Heterogeneous Unlearning Deviation and Skewed Unlearning Deviation and demonstrates them.  To address these issues, the paper  propose FaLW (Forgetting-aware Loss Weighting), an instance-wise dynamic loss reweighting method. The weighting considers model’s current prediction confidence for a sample to the distribution of prediction confidences on unseen data of the same class.  The authors  conduct experiments on multiple image classification benchmarks (CIFAR-10, CIFAR-100, Tiny-ImageNet) demonstrate the effectiveness of FaLW.", "questions": "1.  How does this approach scale to a setup where we want to unlearn a particular class rather than unlearning a particular point from a given class.  \n2. The paper describes FaLW as plug-and-play, but seems to have demonsrated it using only one specific unlearning approach. how can  FaLW can be used in practice an any  generic unlearning approach ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies a novel problem arising in the context of unlearning and proposes a novel solution to address this. The problem addressed is quite relevant and practical. \n\nThe paper  demonstrates empirically the unlearning deviation problem under long tailed distribution setups,  and  defines  the problem clearly, proposing two kinds of unlearning deviation.   \n\nThe proposed FaLW is sound, addressing the identified problem to the extent possible. \n\nSrong empirical results on several real world data sets to demonstrate the effectiveness of the proposed methodology.  \n\nThe presentation of the paper is clear and well written.", "weaknesses": "The methodology addresses the problem to a good extend but suffers from some drawbacks\n\n1. The requirement to have unseen data points from the same class might be impractical.-  in practice such auxulliary data may not be available\n2. FaLW does not provide a formal guarantee or certification that the influence of the forget set is removed \n3. The definition of unlearning deviation in the paper involves a threshold $\\tau_i$, but in the proposed weighing scheme the paper seems to have ignored this. \n4.  The choice of Nornal distribution to model the distribution of predicitive probabailities is not clear.  Why not use a distribution with support in [0,1] which is more appropriate to model distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M28zEcvJY1", "reviewer": "Reviewer_zD1B", "summary": "This paper addresses the long-tailed nature of the forgotten data distribution, which is an important and underexplored problem in machine unlearning. The authors identify two key issues in existing methods: Heterogeneous Unlearning Deviation (HUD) and Skewed Unlearning Deviation (SUD), which leads to biased forgettiing and uneven performance across samples. To mitigate these challegnes, they propose a dynamic loss reweighting strategy that adaptively adjusts learning signals based on forgetting difficulty. Experiments on multiple benchmarks demonstrate that the proposed method effectively reduces unlearning bias while maintaining model utility.", "questions": "1. How easily can FaLW be incorporated into other unlearning frameworks? Would additional tuning or architecture changes be required, or can it indeed serve as a simple plug-and-play loss modification?\n\n2. How do the proposed HUD and SUD differ formally from sample difficulty bias and class imbalance bias that have been widely discussed in prior works? Are there conditions under which HUD/SUD reduce to these known phenomena?\n\n3. Have the authors considered running controlled experiments to isolate HUD and SUD individually (e.g., a balanced dataset for HUD-only, or uniform sample difficulty for SUD-only) to better understand their respective contributions?\n\n4. What is the rationale for the specific dynamic weighting formulation and update rule? Have alternative forms been tested or theoretically compared?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper highlights an under-explored but practically important phenomenon in machine unlearning that the forgotten data often follows a long-tailed distribution. The problem is important and the motivation of the work is clear. \n\n2. The formulation of Heterogeneous Unlearning Deviation (HUD) and Skewed Unlearning Deviation (SUD) provides a structured way to analyze performance degradation in unlearning systems, which offers a useful framing for future work.\n\n3. The proposed FaLW is simple but effective. The paper evaluates multiple metrics and the results consistently show that the proposed achieves better balance between unlearning completeness and retained-task performance.\n\n4. The paper is well-organized and easy to follow.", "weaknesses": "1. Lack of empirical validation for plug-and-play claim: Although the proposed FaLW (Forgetting-Aware Loss Reweighting) is described as a plug-and-play solution, the paper only evaluates FaLW as a standalone framework. There are no experiments demonstrating its integration into other existing unlearning methods.\n\n2. Limited analysis of the identified issues HUD and SUD: The paper identified two important issues: Heterogeneous Unlearning Deviation (HUD) and Skewed Unlearning Deviation (SUD) as key motivating factors. Howerver, these notions closely resemble exiting sideas such as sample difficulty bias and class imbalance bias from the broader learning literature. The paper does not sufficiently differentiate its definitions from these established concepts, nor provides diagnostics or ablations to examine HUD and SUD independently.\n\n3. Lack of isolated analysis for HUD and SUD: It remains unclear whether HUD and SUD always co-occur or can arise independently. The experiments treat them as jointly existing phenomena under the long-tailed setting, but no analysis is presented to determine their individual effects on unlearning performance.\n\n4. Lack of discussion on design choices and hyperparameter sensitivity: While the method introduces a dynamic loss reweighting mechanism, the paper provides limited justification for specific design choices, like the form of the weighting function,or the dynamic adjustment rule. A more detailed discussion on hyperparameter selection, or sensitivity analysis would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gdrUxSUs7e", "reviewer": "Reviewer_N7tQ", "summary": "The paper introduces FaLW (Forgetting-aware Loss Weighting), a novel loss design intended to improve the reliability and controllability of machine unlearning. The key idea is to dynamically reweight forgetting and retaining objectives based on the model’s forgetting confidence and feature-space similarity, thus preventing over-forgetting and instability observed in prior works. The authors derive an adaptive weighting function grounded in information-theoretic uncertainty and validate their approach on several image classification benchmarks using standard unlearning baselines.", "questions": "See the comments above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is, to the best of my knowledge, the first to explicitly formulate long-tailed forget sets (not long-tailed training data) and to show that existing approximate unlearning methods exhibit heterogeneous and skewed unlearning deviations under this realistic setting. This is an underexplored but practical scenario.\n2.\tThe proposed FaLW is conceptually simple, instance-wise, and orthogonal to most gradient-based unlearning pipelines. It can be adopted with minor code changes.\n3.\tThe direction-aware weighting derived from per-class unseen distributions provides a principled way to decide whether to increase or decrease forgetting pressure for each sample, which directly matches the identified deviation phenomena.\n4.\tThe paper is well-written and logically consistent, with clear motivation method experiment alignment.", "weaknesses": "1.\tLimited theoretical justification – while the adaptive weighting function is motivated by uncertainty, the derivation remains heuristic. The paper lacks formal analysis or convergence guarantees explaining why the proposed weighting yields more reliable unlearning.\n2.\tAblation insufficiency – although the paper reports a few ablations, it does not disentangle the specific contributions of the uncertainty term versus the similarity term in the weighting function.\n3.\tLack of comparison with recent conformal or calibration-based approaches – given the growing body of work, FaLW should also be compared in terms of uncertainty calibration and reliability metrics to position itself clearly.\n4.\tPotential instability in extreme regimes – adaptive weighting can introduce oscillations or under-forgetting when uncertainty estimates are unreliable, but the paper does not report sensitivity or failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTb2MKp624", "reviewer": "Reviewer_PHd4", "summary": "FaLW addresses machine unlearning when forget sets are long-tailed. It quantifies instance-level unlearning deviation by comparing a sample’s confidence to unseen-class probability distributions, then reweights forgetting loss with a class-aware balance factor, mitigating over-forgetting and under-forgetting and improving accuracy and MIA performance.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n\n1. This paper first works on the long-tailed forgetting data unlearning\n2. This paper proposes a new plug-and-play instance-wise unlearning method", "weaknesses": "Weaknesses:\n\n1. This paper lacks an explanation of the differences between the unlearning on the general datasets and the long-tailed datasets.\n2. This paper assumes that the class-conditional prediction is a Gaussian distribution, which is too strong. This paper lacks the validation of this key assumption\n3. This paper does not conduct an effective ablation study for each component in Eq. 3.\n4. The results do not contain std\n5. This paper only uses VGG16 for experiments. More models should be tried.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)