Sharpness-Aware Machine Unlearning
Authors: Haoran Tang
Rajiv Khanna
Venue: ICLR 2026 Poster (2026)
Keywords: Machine Unlearning
Sharpness-Aware Minimization
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We reveal the effectiveness of sharpness-aware minimization in machine unlearning; we extend existing theory to provide refined characterization of SAM in unlearning, and propose novel algorithm based on detailed empirical results.

Abstract: We characterize the effectiveness of Sharpness-aware minimization (SAM) under machine unlearning scheme, where unlearning forget signals interferes with learning retain signals. While previous work prove that SAM improves generalization with noise memorization prevention, we show that SAM abandons such denoising property when fitting the forget set, leading to altered generalization depending on signal strength. We further characterize the signal surplus of SAM in the order of signal strength, which enables learning from less retain signals to maintain model performance and putting more weight on unlearning the forget set. Empirical studies show that SAM outperforms SGD with relaxed requirement for retain signals and can enhance various unlearning methods either as pretrain or unlearn algorithm. Motivated by our refined characterization of SAM unlearning and observing that overfitting can benefit more stringent sample-specific unlearning, we propose Sharp MinMax, which splits the model into two to learn retain signals with SAM and unlearn forget signals with sharpness maximization, achieving best performance. Extensive experiments show that SAM enhances unlearning across varying difficulties measured by memorization, yielding decreased feature entanglement between retain and forget sets, stronger resistance to membership inference attacks, and a flatter loss landscape. Our observations generalize to more noised data, different optimizers, and different architectures.

Published as a conference paper at ICLR 2026
SHARPNESS-AWARY MACHINE UNLEARNING
Haoran Tang
Rajiv Khanna
Department of Computer Science Purdue University
Department of Computer Science Purdue University ra rajivak edu
thr@purdue edu
ABSTRACT
We characterize the effectiveness of Sharpness-aware minimization (SAM) un- der machine unlearning scheme, where unlearning forget signals interferes with learning retain signals. While previous work prove that SAM improves general- ization with noise memorization prevention, we show that SAM abandons such denoising property when fitting the forget set, leading to altered generalization depending on signal strength. We further characterize the signal surplus of SAM in the order of signal strength, which enables learning from less retain signals to maintain model performance and putting more weight on unlearning the forget set. Empirical studies show that SAM outperforms SGD with relaxed requirement for retain signals and can enhance various unlearning methods either as pretrain unlearn algorithm. Motivated by refined characterization of SAM unlearn- ing and observing that overfitting can benefit more stringent sample-specific un- learning, we propose Sharp MinMax, which splits the model into two to learn retain signals with SAM and unlearn forget signals with sharpness maximization, achieving best performance. Extensive experiments show that SAM enhances un- learning across varying difficulties measured by memorization, yielding decreased feature entanglement between retain and forget sets, stronger resistance to mem- bership inference attacks, and a flatter loss landscape. Our observations generalize to more noised data, different optimizers, and different architectures. Our code is wttp://thw...ooo.aaaaaaaaaaaaaae hann///aaaaaanangg/arruuumnnn
available
INTRODUCTION
Deep neural networks have grown so large and complex that retraining a model from scratch to forget even a few samples has become impractically costly in both computation and energy. This challenge has catalyzed the study of machine unlearning: methods that efficiently remove the influence of specific training data without full retraining, aiming to forget designated examples while preserving overall performance. Numerous unlearning strategies have been explored - from influence-based updates that subtract a data point's contribution (Izzo et al., 2021), to fine-tuning with targeted weight sparsification (Jia et al., 2023), to joint optimization approaches that explicitly balance "retain" vs. "forget" objectives by gradient ascent/descent However, a fundamental understanding of what makes unlearning effective remains elusive. Key questions persist: How should we trade off forgetting unwanted data versus retaining accuracy on the rest? How do different training algorithms influence unlearning dynamics? Why are some samples inherently harder to forget than others? In practice, the lack of principled answers has led to ad-hoe hyperparameter tuning and unpredictable behavior across tasks. In particular, when a model is simultaneously fed with conflicting retain and forget signals, these signals can interfere and even cancel out during training. hampering the unlearning process (Kurmanji et al., 2023). To date, there are few robust solutions to mitigate this interference, underscoring the need for a deeper theoretical foundation for machine unlearning.
different data subsets (Kurmanji al., 2023).
Recent advances in learning theory and optimization hint at possible directions to tackle these issues. First, Signal-versus-noise perspective has provided new insight into model behavior: for example, Chen et al. (2023) formalize how networks learn meaningful patterns while ignoring or memorizing label noise, and Zhao et al. (2024) empirically identify factors that make certain data points harder to forget. Particularly relevant is the Sharpness- Aware Minimization (SAM) method (Foret et al., 2020) that has been shown seek flatter loss minima and thereby dramatically reduce memoriza-

---

Published as a conference paper at ICLR 2026
tion of noisy data, leading to improved generalization in noisy-label settings (Chen et al., 2023). These observations suggest that a model's ability to distinguish true signal from noise may be key to effective unlearning. An optimizer that naturally suppresses memorization of noise might also be better suited for forgetting specific examples when required. To investigate this hypothesis, we quantify each sample's memorization level using established metrics (Feldman, 2020; Feldman & Zhang, 2020), allowing us to rank the "forget set" by difficulty. This enables a controlled study of how different optimization algorithms perform when asked to forget data that the model has learned to varying extents.
We present a comprehensive theoretical and empirical study of machine unlearning through the combined lens of signal-noise decomposition and sharpness-awar optimization. We focus on the challenging scenario where both retain and forget samples are present in each training batch with mixed objectives, and we compare standard Stochastic Gradient Descent (SGD) to SAM in this con- text. Building on recent theoretical frameworks for ReLU networks (Kou et al., 2023), we derive rigorous results for two-layer CNN that characterize the unlearning process under each optimizer. Our analysis yields several striking findings. (1) SAM's noise suppression can break down under unlearning: we prove that when tasked with intentionally forgetting set of samples (treated as "noise"), SAM is forced by objective to abandon its usual denoising behavior effectively over- fitting the forget set nearly as much as SGD does. This result challenges the expectation that flatter-minima methods would inherently excel at unlearning. (2) We establish formal guidelines for factor
balancing retain vs. forget objectives: in particular, we derive the minimum retain-weightte needed to prevent catastrophic forgetting of the kept data. Our theory shows that SAM can accom- plish successful unlearning with a significantly smaller retain weight than SGD, meaning SAM tolerates a stronger forgetting signal without sacrificing retained accuracy. In the regime of benign overfitting (where the model fits even noisy data without large generalization error), we quantify the gap in required a between SAM and SGD and prove it scales on the order of O(vd/n) (with d the model dimension and the training set size). (3) Perhaps most surprisingly, our findings call for a re-examination of overfitting in unlcarning. Contrary to conventional wisdom, we show that deliberate overfitting- in a controlled way that limits its impact on the rest of the data can enhance the complete removal of those samples. This insight is especially relevant in stringent privacy or copyright scenarios, suggesting that the strict avoidance of overfitting may not always be optimal.
Our contributions can be summarized as follows:
We introduce rigorous analytical framework for machine unlearning based on signal-noise decomposition. This framework explicitly models the interplay between retain and forget signals. Using this lens, analyze the behaviors of SGD versus SAM and prove that SAM's denoising advantage "shuts off" on forget data: when SAM is asked to unlearn labeled noise, it ends up overfitting to the forget set almost as much as SGD.
Theoretical Framework:
Balancing Retain vs. Forget Objectives: We derive provable guidelines for balancing the re- tain/forget trade-off. In particular, we identify the minimal value of the weighting ratio parameter a that guarantees sufficient retention of knowledge. We show that SAM requires strictly smaller a than SGD to achieve effective unlearning. In the regime of benign overfitting for both the optimizers, we analytically bound the difference in required a on the order of O( /d/n).
Through extensive experiments on CIFAR- 100 and ImageNet datasets, we validate our theoretical insights. We demonstrate that incorporating SAM into state-of-the-art un- learning methods consistently boosts forgetting efficacy while better preserving accuracy on the re- maining data. Models optimized with SAM yield flatter loss landscapes and reduced entanglement between retained and forgotten samples, corroborating our theory that SAM distinguishes signal from noise better. We also observe that SAM-trained models are less vulnerable to membership inference attacks to forget set, indicating improved unlearning.
Empirical Validation:
Finally, inspired by our analysis, we propose Sharp MinMax, a new unlearning approach that decouples the retain and forget objectives. Sharp MinMax splits the model into two cooperative parts: one is trained with SAM on the retained data, while the other performs sharpness maximization the forget data intentionally overfit those samples to ensure forgottenness. This design mitigates interference between retain and forget signals. Sharp Min- Max achieves state-of-the-art unlearning performance in our experiments, especially on challenging
Novel Unlearning Algorithm:
2

---

Published as a conference paper at ICLR 2026
high-ment -igh-memorzzatonn forget sets, where it significantly outperforms existing techniques in completely erasing the target data's s influence.
PRELIMINARIES
2.1 DATA AND MODEL. CONSTRUCTION
We construct a practical learning scenario which distinguishes between useful and unrelated signals from inputs. Similar constructions have been adopted previous work (Kou al., 2023; Chen et al., 2023) with rich notation. For convenience, we summarize a table of notation in App. C. Consider learning binary classification with label y E {+1} using a two-layer CNN on image training data set S {(x,, 24 NE(M) D. Each image consists of P patches and assign randomly of them the signal yiςp for label y; and the universal signal vector 4 Rd. and represent other patches by the noise vector & E I' ~ MO0, of.). Thus, each input image is vectorized as X, E. ................................ yrP ..... E,] E Pxd, where yp can appear at any position.
The second layer of CNN is fixed as +1/m respectively for m convolutional filters. The two-classes network can be expressed as f(W, x) f+1((+WWWW) (W+1, - f-1(W-WWW-- -1,), where
m
m
(W x)
a ((w (. x))
o ((w + (P 1)o((( E))- (1)
m
m
p=1
Here a denotes ReLU activation, War E Rd denotes the weight for the r-th filter, and W, is the collection of model weights for j #1. We train this CNN with cross-entropy loss C(W,S). Denote w (t.b) for j E (#1},r= r E [m]] the convolutional filter at the b-batch of t-th epoch of SGD. j,r
(t.b) ((t.b) for learning
We decompose the weight update into learning signal and noise coefficients ne jir the signal and the noise respectively, such that
W (1,b)
(0.0) + jjr 00ااe1 + (P-1)
(t.b)
(1.6) Sun
E
(2)
(3.b) and decrease S This construction also extends to where the learning goal is to increase Kj.r multiclass classification considering one vs. all setting with binary classification problems. For readability, abbreviate subscript j, r and replace superscript (t, b) with time vector in following sections, and leave full notation to proofs in the Appendix.
2.2 SIGNAL-TO-NOISE UNLEARNING
Given a pretrained model fTh by algorithm A for Ti epochs on S, machine unlearning aims to eliminate the influence of forget set F S to the model training. while maintain generalizability to unseen data without compromising performance on the remaining retain set R = S F. Denote the unlearned model as fi by unlearning algorithm U, which is initialized as TT and unlearned for T2 epochs. We consider unlearning small portion of S with much less expense than retraining the model from scratch on R, so F < |R| and Tr < Ti.
Random Label (RL) (Graves al., 2021) aims to unlearn by finetuning on S but with F's labels randomly flipped in each epoch. It naturally fits into our setup as fabelfflipeed F become the noise, and motivates us investigate unlearning algorithms under the same theoretical framework. The gradient update of at and Ç of class j can be expressed as
t-1 nt nllspl13
Σ ('o' Tisp)) - Eeto'(ww ((W'
Bm
LEER
BEIE
(3)
c. c. ((P
e:t "'o((w" ,i)) sgn(y; = j),
Bm
where B, denote the batch size and learning rate, sgn(-) denotes +1 sign function, IR and IT denote batch samples from R and F t, respectively. In each iteration, I aims to erase its signal in rst while &, reinforces or decreases ç update depending on label agreement.
3

---

Published as a conference paper at ICLR 2026
Negative Gradient (NegGrad) (Kurmanji et al., 2023) unlearns F using gradient ascent while gradient-descending on R. Unlike RL or other U that aim at random guessing, aacent-based un- learning encourages misclassification by its objective:
NeggGra R, =
1 R
Σ
al (wif (W,x))) ER
[u-a)((u((W,)))
(4)
|
EEF
Minimizing LNegGrad induces competing gradients, canceling each other during K; C update. serves as a weight coefficient that accounts for the size imbalance between R and F. To synchronously optimize the model with retain and forget samples, we draw B samples from both subsets each batch and train for [R|/B batches. Thus, forget samples' signals are relatively enlarged by a fraction of due repetition. Heuristically, x [R|/([F] [R]).
[R|/ [
2.3 DENOISING PROPERTY OF SAM
Sharpness-Aware Minimization (SAM) (Foret et al., 2020) aims to minimize perturbed empirical loss the worst point in the neighborhood of W, solving the following optimization problem:
min ((W, S) + max L(W + +2 ((W,S)
for controlled perturbation ?. It ensures uniformly low training loss and avoids sharp landscape. While both SGD and SAM learn a sufficient signal with x² = S2(1) after Zi epochs, Chen et al. (2023) prove that SAM outperforms SGD by noise suppression and SAM upper bounds <?' by ((1) while SGD dimension dependent (log d). The key difference stems from the noise memorization prevention of SAM. Given the perturbation term E in SAM for class j:
I Σ IE PE[P]
i y:o' ,
Xxpp))x VWw I.)
(6)
consider ReL.U activation at any fixed iterate w² for SGD: (w* Ex) 2 vss. SAM:(w* + E. Ex) for I,. yk- SAM's (w* + +,,, +xt)))) expands to (w*, E&) + (E) Ex), where (2), Ex) is proven to be sufficiently negative to cancel (w*,Es) by selecting a proper T, thus deactivating the noise (Chen et al., 2023). This effectively prevents SAM from learning from the noise which would lead to harmful overfitting for SGD. We are curious about whether SAM improves unlearning: a flatter landscape can make learning easier, then it should make unlearning easier too despite a reverse sign. But is it simple adaptation, and can we straightforwardly extend previous theories and findings to develop unlearning algorithms?
THARPNESS-AWARE UNLEARNING
We first show that the SAM's noise memorization prevention in Sec. 2.3 does not fully hold when SAM is used with NegGrad for gradient ascent F. Specifically, SAM overfits forget signals as much as SGD, while maintaining its denoising property on R. Based on this result, we derive refined test error bounds for SGD and SAM under NegGrad and characterize the different thresh- olding between SGD and SAM for unlearning. Although SAM continues to improve unlearning and maintain generalizability, the altered activation patterns and unlearning behaviors by previous works, as SAM is forced to fit forget signals (viewed as noise) by NegGrad objective. This leads divergent behaviors on R and which can be of independent interest.
captured
3.1 NEGGRAD REVISITED
Unlike RI., the mutual interference between F and R under NegGrad additionally affects Ç update. The update rules for tt and ç under NegGrad now become:
nl Bm
kt xe
pp, (1 (1-a)
pp
IEIR
IEI7
'(P- 1)2
c1+] -²
Ve. - - (1-a) Σ Ve
Bm
SEIR
1EI!
4

---

Published as a conference paper at ICLR 2026
d'ww"' + 6, Ei)), and 5 2* for SAM and for SGD. In plain words, a retain sample of class causes a decrease in Cj. discouraging memorizing noise for the correct class, while another retain sample of class -- causes increase in Gj. encouraging Wy to use & to distinguish class j from -j- Conversely, a sample E F of class j, which we want predict -j in ascent-based unlearning, will increase çj and encourage wy to use noise & in way that harms class j. and vice versa. Similar intuition also applies to k. The interference in Ç update will alter SAM's behaviors towards forget signals as summarized in Lemma 3.1.
where Ve:; = 2 5, yipp)) Vg: sgn(y; 5)زز, [24
Lemma 3.1 (Noise memorization of by SAM under NegGrad). Under the NegGrad scheme and the Assumption D.I holds. for class j we have that if for SGD: (w*, Ex) 2 0. k I and j yk. then for SAM: (we + ,,,,x) < 0. However, for SGD: (w* Ek) 2 0, k E I and = then for
SAM: (w² + E. Ek) > O.
See proof in App. D.2. Because the activation patterns on T and I diverge, SAM continues to suppress noise memorization and leverage its sharpness-awann aware updates when fitting R. but "falls back" to SGD-like behavior on F. This split yields two distinct sets of bounds on and Ç for R and F, which lead to separate test errors shown in App. D.1 and D.2. However, given a pretrained model TT with HT > to start unlearning, as long as retain signals weighted by a dominate, the signal strength will remain sufficient and continue to grow. This is shown in Chen et al. (2023) when the signal strength is saturated at T < Ti. We can thus choose a threshold based on this principle. With proper forget-retain size ratio, results in Chen et al. (2023) still hold: SGD's test error converges when signal strength is sufficient, but can't 't be upper bounded otherwise; SAM's test error converges either way. B3 serves as a knob to control the convergence rate:
Theorem 3.2 (SGD test error under NegGrad). Under Assumption D.I, for any > 0 and 1> a 2 [R]/(FF + (R]) = 3 > 0.5, then with probability at least 1 - 5, the training loss converges: L(WT, (WW",) D) I E. Moreover, if 1|p1|2 2 C2d aad//An1//Pop Po. we have the test error L (WT, D) S c. If || S C2dl/'n n/1///nneppp Pop. we have lim L (eet D) 2 0.1, and limg- L (WT, D) 2 0.05.
Theorem 3.3 (SAM test error under NegGrad). Under Assumption D.I, for any e >> O and 1 >> 2 [R]/(FFI [R]) B > 0.5, choose O mayya). Then with probability at least 1 8, the training loss converges: L(WT, , D) S 6. Moreover, if |||||12 2 -/dt/An-1/44
we have limg-1 Liest (WT D) S E. If 9(1) I 11l122 I C2d -
P/p we still have
lim L "ee"(W"' ( E.
See proofs in App. D.1 and D.2. Together, these theorems describe how SGD and SAM behave when retain signals dominate. For SAM, if ||1/|22 & Czdl/An-1i Pap. p. it will suffer harmful overfitting to F. However, long as 2 [[F + [R]) and ||0||2 1((1), learning on R guarantees overall benign training and yields a bounded test error. Under the same condition, Corollary 3.3.1 concludes that while the signal coefficient continues to grow for both SGD and SAM, SGD's noise accumulation is loosely bounded by model dimension, while SAM's by (())
Corollary 3.3.1 (r., S update under NegGrad). Under the NegGrad, i Z |R|/(FF++ + (R)), since R = R(1), both SGD and SAM continue to grow. Given the learned ÇT), SGD continues to overfit the noise with Oflog d), while SAM overfit the noise from with Ollog d) and from R with ((1).
See proof in App. D.3. Finally, we characterize the differed choice of for SGD and SAM SAM learns signal more efficiently. We also reveal that a depends not only on forget-retain size ratio as commonly conjectured, but also on the signal strength, and thus the dimensionality of the problem:
Lemma 3.4 SSgnall-urpluss of SAM under NegGrad), Under the NegGrad, for any &p where || 2 2(1), SAM exhibits faster signal learning on R.: ASAM k./ A spuch K = 0(1p 13).
See proof in App. D.4. As a result, SAM relies on a more relaxed a threshold than SGD due to faster signal learning. For SGD to achieve the same signal learning performance as SAM, we need to scale up SGD to satisfy sSGD asGD/a SAM @11(p)))) If ||</||2 2 0,dt/4n11/ Pa,, and both SGD
5

---

Published as a conference paper at ICLR 2026
and SAM achieve benign overfitting, then given the extra signal learning from R, SAM results in faster update and surplus signal of 1R/1/ in each unlearning epoch.
3.2 SHARP MINMAX
In Sec. 3.1, we showed that SAM is provably better of sample test errors under NegGrad, and we empirically verify that SAM achieves better unlearning performance in Sec. 4. But how does the refined characterization matter, given maintained test error conclusions? Jointly with em- pirical observations, the altered behaviors of SAM on F motivates new unlearning algorithms. Our experiments show that SAM+NegGrad attains higher forget accuracy than SGD+NegGrad, forget- ting less effectively. This finding forces us to reconsider the conventional view that overfitting is always detrimental: while overfitting indeed harms generalization, it may be beneficial when the goal is to remove specific samples from model. Consequently, for abstract concept forgetting we continue to demand strong generalization; but for stringent scenarios where exact sample removal is mandated by privacy or compliance constraints model's tendency to overfit can actually en- hance its unlearning of those exact points. The divergent behaviors under SeLM+NegGrad motivates the following new algorithm: F, denoted as the forget model W,, while leaving the rest as the retain model W R. to maximally maintain the model utility by leveraging SAM purely R. Motivated by how SGD with sharper minima tends to forget better, we propose Sharp MinMax to intentionally optimize for sharper-than- SGD minima with the purpose of overfitting forget signals for unlearning. Inspired by Kim al. (2023), we leverage sharpness maximization on W
split portion of model parameters to purposefully overfit to
min C(W F) max C(W, y + 2,F) C( ,. WF
(8)
resulting in sharper landscape that harms the generalization by overfitting. We apply weight mask- ing based on gradient magniudes (Fan et al., 2023) to divide our model into WR. W during optimization. Specifically, we pass F to fA once, accumulate gradients for each parameter, and check top parameters with smallest magnitudes cut off by a given percentage. We then apply SAM W,. The retain model with SAM is already characterized
on WR and sharpness maximization by Chen et al. (2023), while W F requires a stronger signal strength than SGD to avoid harmful overfitting, See implementation details in App. E.2.
3.3 QUANTIFYING UNLEARNING DIFFICULTY WITH MEMORIZATION
We examine the effectiveness of unleaming UI based on memorization, which sufficiently reveals the difficulty of unlearning (Zhao et al., 2024). Feldman & Zhang (2020) define the degree to which sample memorized by a pretraining AA on example (x;, from as the memorization score:
mem (A, S,i) == Pr [f (W.,), = =yi f-A(S\i) f-((())
Pr (W, = y.l.
where S i denotes S with the sample (x;, yi) removed. Samples of high-memorization scores can be atypical samples which model usually learns later in the training process after more updates the model than typical ones. Thus unlearning them would be harder and may require more iterations of unlearning steps which may impact the model performance on the retain distribution. The converse is true for samples of mommmimoiinttion scores. We can hence construct FF of varying unlearning difficulties based on memorization scores to comprehensively evaluate 24
4 EMPIRICAL STUDY
We conduct major experiments on CIFAR- 100 (Krizhevsky et al., 2009) and ImageNet- 1K (Rus- sakovsky al., 2015) using ResNet-50 (He et al., 2016), and adopt pre-computed memorization scores for from Feldman & Zhang (2020) to generate F of different difficulties with |F] R 5% S) denoted as [Frigh, ; Fiew]. For both pretraining and unlearning, we adopt SAM (Foret et al., 2020) with p = 0.1 and Adaptive SAM (ASAM) (Kwon et al., 2021) with p = [0.1, 1.0]. We ensure same optimal syper-paprameters for each comparable [SGD.SAM] pair. See details in App. E.
Evaluation. We follow previous work (Triantafillou et al., 2024; Zhao et al., 2024) to measure the wfg-of-war tradeoff between forgetting and retaining of fu based on accuracy Acc(O, D), with the retrained model fA(R) as reference:
(fw( =(1 - (Aco(f R) - Acc(fu, R))) (1 -A) -cc((frrR) F)) (10) (1 - (Aco(fA(R)) Duaa) - Acc(fu, Drea)) with test transforms on R.F.

---

Published as a conference paper at ICLR 2026
Table 1: ToW(%) (%) of unlearning on ImageNet- IK and CIFAR-100. For each (U, pair, we report ToW of each F and compute averages. SAM consistently improves current unlearning methods.
ImaggeNet
-SGDD
1-ASM 6.1
4-ASAM 1.0
Unleam
High
Law
AVG
11ihh
Mid
Low
AVG
1iihh
Mid
Low
AVG
1kgh
Mid
Low
AVG
Neglirad
78.764
84. 190
68.515
$3.826
78.436
53.93
96.651
83. 002
78.522
83:029
50.55 947
54.133
78.03
94.176
68.890
63
78.52
84. 14.
89.188
83.94
78. 386
$4.07
80.098
83. 845
78.762
84.267
90.579
$4.536
73.053
84.062
89.973
84.009
+SSA 1.D
78.966
83.389
92.176
$4.843
78,975
63.358
91.843
64.725
78.027
83.326
92.772
$4.708
77.762
K3.261
92.617
84,554
+SAM
77,898
82.965
92.841
$4.575
78,301
53.04
91.722
64:354
77.368
82:473
93.429
$4.43
76,807
82.567
92.829
84.074
RL
74.598
86 61.
86,714
$2.643
74,857
86,652
96.192
82. 504
74.317
86.813
$7.670
$2.93
74.055
96.715
68.594
83.121
+ASAM 1.
74.95]
85.
91.069
$3.867
75.231
83,473
90:425
83.707
73,950
85.393
91.516
83.62
73,579
85.494
91.74
R3.604
SalUi
44.911
71.889
95.008
70.609
46, 104
71.735
94.652
70.85
458814
72.308
95.316
71,077
46.006
72.419
95.215
71.214
+ASAM 1.0
45.998
71,554
95.628
71.06
46.908
71,268
95,224
71.143
45,856
71,695
95.924
71.158
46.358
72.034
95.791
71,394
CIPAR100
SSGD
4-ASAM 0.1
ASAAMM
4-SAM 0.1
Unkern
Mid
High
AVG
High
Mai
AVC1
NegGrad
78.334
83.335
85.718
$1.796
19,217
96.154
S5.637
84.789
77.274
76.59
55.443
50.436
67.826
74.145
76.374
72.78
+ASAM 0.1
78:131
82.845
96.75
82.586
50.336
51.539
$7.671
85. 182
77.351
79.074
$9.009
$1.452
70.054
74.158
78.087
74.1
AASAM 1.0
80800
X1.465
87,052
83.108
62.196
84.591
90.502
R5.66%
76.781
79.30 264
93.249
$3.748
72.518
75.653
86.759
71.31
4SAM 0.I
81.331
75.059
94.151
83,514
92.56
77.94
94.179
84.993
74.704
70.548
95.898
80.5
63.050
66.089
95.075
75.416
LI-Spaeec
63.445
68. 685
53.991
62,042
63.099
72.775
60.34
65. 6115
61.252
68.197
61.47
63.64
65.258
71.941
59.014
65 655
+ASAM 1.0
66.903
75.554
38.967
67.341
66.213
77.119
66.657
70.11
65.117
73.754
62.517
67.129
63.151
74,556
65.117
07.575
SCRUB
58.415
76.125
12.70%
49.084
67.163
79.00
10.823
52. 350
57.816
75.176
55,483
63.15X
43.246
06.433
17.365
43.016
+ASAM 1.0
50.313
73.353
97.631
73.766
00.515
90.204
97.508
79.401
48.569
73.09
97.776
73.145
18.137
03.615
97.933
59.229
RL
68:164
$4.395
72.4
75.086
64.518
50.215
00.711
71 481
66.689
86.411
69.677
74,259
64.391
85.46
70.55
73.474
4ASAM 1.0
69.522
M6.779
74.409
77.047
66.909
56.557
09.375
74.240
69.75
91.124
3013311
$0.392
72,854
55.033
78.066
792861
SalUn
69.926
83.005
71.75
74.904
66.541
83.577
71.95
73: 956
67.355
89,700 768
79. 005
78, 738
69.671
90.695
75.28]
78.482
+ASAM 1.0
73.268
02.225
88.175
$4.556
71.426
89.152
56.13
82.246
67.715
93.401
89,289
$3.468
70,933
92.914
86.477
X3.441
Table 2: MIA (%) conrectness to F on CIFAR- 100. observe consistent improvement.
We enhance each U with ASAM 1.0 and
A -SGD
A B.
ASAM 1.0
.4 4-SAM D.
Unteamn 12
Tlgh
Mi4
Low
AVG
Hlgh
Mid
Low
XVG
10gh
MBd
Luw
AVG
1ligh
Mid
Low
AVG
94:733
61,233
94.933
51.433
03.833
62.067
5.8
92.867
60.033
5,033
6944
+ASAM 1,0
94.267
38.5
5.5
52.756
94.3
55.3
3.5
51.733
93.633
56.033
3.9
$1.189
93.8
59.333
3.8
$2.311
SCRUTR
55433
18.6
53.6
64.733
23.1
71.613
53.155
54.767
16.133
9.833
36.911
39.3
9.333
56.3
1444
+ASAM 1.0
46.467
14,867
0.1
20.478
$7.367
22:633
0.167
26.722
44.7
14,557
0.2
19,822
14.433
2.333
0.2
5.655
RL
90.367
62,933
10.767
54,822
91.633
68.267
13.
57.8
89.067
63,567
15.8
56.145
59.157
61.967
8,267
53.134
+ASAM 1.0
90.3
61.3
9.467
$3.689
91.5
62.667
12.7
$5.656
$5.0
61.3
10.667
53.322
36.3
59.833
3.833
50.655
SalLis
83:433
59.233
7,335
50.0
84,533
59.1
11.167
51.6
79.3
54,667
8.8
47.589
$1.457
53.133
6,867
47.156
+ASAM 1.0
79.1
51.833
4.5
45.144
$1.7
$6.167
6.633
47.50
74.967
49.5
4.2
42,889
75.633
47.667
4,067
42.456
NegGrad
86.933
37,233
2,167
42.111
58.867
40.2
1,733
43.60
82.167
32.1
1.8
38, 689
74.657
36.967
3,433
38.356
+ASAM 1.0
84.5
30.1
0.738
38.444
$5.5
30.1
0.7
38.8
81.233
24.533
0.583
35.433
73.967
20.733
0.366
31.689
Thus, we encourage high retain/test accuracies and low forget accuracy. Note that our ToW differs from that in previous work as we measure the raw accuracy difference instead of the absolute differ- ence, because new unlearning methods that continue to fine-tune on R can outperform S'(R)) within a conventional unlearning time T2. If using the absolute ToW, a higher test accuracy than SA(R) will be penalized and the model performance cannot be properly measured.
4.1 SAM CONSISTENTLY OUTPERFORMS WITH BETTER TRADEOFF
We conduct unlearning with various unlearning algorithms 2 given different pretrained f.A- Tab. reports ToW scores of on CIFAR-100 and ImageNet. We observe that SAM consistently improves all unlearning methods under different initializations f suggesting that SAM can universally enhance prevailing U. While different t exhibit varied effectiveness to [Frigh- Fuid, Fow). we observe that NegGrad achieves a better balance between three forget sets than other methods. We include detailed [retain, forget, test] accuracies, further analysis and demonstration of statistical significance in App. F. Upon close examination on those accuraices, we observe that despite SAM outperforms SGD by better retain and test accuracies and thus better ToW, SGD can oftentimes achieve lower forget accuracies. This aligns with our theoretical analysis where SGD overfits more to F, and also sparks our Sharp MinMax. Smaller experiments on CIFAR- 10 and Imay-ea in App. G yield aligned conclusions.
MIA correctness. We report correctness rates of membership inference attack (MIA) to F on C00 in Tab. 2. Lower correctness means better unlearning: forget samples behave more like samples that were never in S. We find that SAM consistently improves data privacy while unlearning more effectively. Note that NegGrad achieves better MIA correctness than RL: this is because gradient ascent actively erases gradient signatures of JF in the model. SCRUB (Kurmanji et al., 2023) with SAM achieves best MIA performance.
7

---

Published as a conference paper at ICLR 2026
Relearning attacks. We also present relearing attack experiments to demonstrate SAM's unlearn- ing robustness in App. G.5. We observe that SAM enhanced U are more resilient to relearning at- tacks with smaller increases. While not main focus, these experiments highlight the robustness of our approach and encourage future works for deeper investigation into the role of loss landscape geometry for robust unlearning.
KL on margins. While ToW measures performance closeness between unlearned model and re- trained model, Georgiev al. (2024) propose to measure distribution closeness in output space with KL divergence on margins (KLoM). We evaluate NegGrad w/ SGD vs. w/ SAM on CIFAR-100 using KLoM means and 95%-percentiles (tails) in App. G.7, and observe similar conclusions: on KLoM means, SGD can outperform SAM on F. but SAM achieves better closeness on R, Dest and hence better KL.MS overall; on p5%ceeercetiles. SAM outperforms SGI) for all settings, suggesting lower variance and better stability at tails. While SAM is not targeted to resolve data dependency issues in unlearning (Georgiev et al., 2024), it ameliorates them by its geometric properties as sug- gested by lower entanglement and better ToW and KLoM.
Our observations further generalize. We consider structured noise unlearning. where another source of noise is introduced during unlearning. We adopt the glass blur and snow effect from ImageNet-C (Hendrycks & Dietterich, 2019) to corrupt R and F of CIFAR- -0FAR-100,, and unlearn with NegGrad and Sharp MinMax. We record experiment results in App. G.3, and observe consistent con- clusions where SAM outperforms under both corruptions. We also experiment on ViT--mall (Doso- vitskiy et al., 2020) with AdamW (Loshchilov & Hutter, 2017) on CIFAR-100 in App. G.4, with NegGrad and Sharp MinMax. We continue to observe promising improvement by adding SAM, with significant increase of ToW on Sharp MinMax. While we focus on studying the geometric properties of SAM rather than efficiency, we are the first to demonstrate how recent efficient SAM variants (specifically Momentum SAM (Becker et al., 2024)) can perform equivalently well as the original SAM with much less computation overhead in App. G.6.
4.2 CONSTRAINED OVERFITTING BENEFITS UNLEARNING
Table 3: ToW (%) t of Sharp MinMax on ImageNet- IK and CIFAR-100. Comparing with Tab. 1, Sharp MinMax achieves new best ToW performance.
ImageNet Unicom
A
ASSGD
A 0.
AASAM 1.0
0,
Figh
Mid
Luw
AV0
H1gh
Mid
Law
A00
1ngh
Mid
Liw
AVD
Hlgh
Mid
Low
AVG
23.357
30.881
86.334
73,418
80.774
54.37%
79.527
73.103
$1.105
86.402
80.913
79,827
ASAM 0.
78.066
87.914
87.338
86.84
79:077
87.4
96 933
84.475
70.148
$8.039
87.554
81.914
78.529
87.642
56 868
84.26
ASAM 1.0
86
87.345
87,899
86.166 166
892
89. 138
87.408
$6.915
$7.27
142
872272
87.076
50.064
87.804
SAM 0.1
86.463
86.755
90.005
87.741
83.511
86.635
89.832
87.333
36.849
36.722
91.111
$8.227
55.712
36.486
90.200
87.468
CIFARIIO
A4 SSGD
_4 ASAA 0.1
A ASAM 1.0
A -SAM 0.1
Unnearn I1
High
Mai
AVC
High
Mid
Tyme
AVG
High
Mrd
AVG
High
MM
AV2I
SGD
70.7664
76.602
82.853
76.771
73.137
77.864
81. .17
77. 28.2
65.925
74.536
80.127
73.526
60.478
71.931
73,845
68.751
ASAMOL OLI
78,99
96.027
83.473
86. 132
64.968
96.451
82. 583
88. 101
$1.825
93. 786
87.151
87. 397
72,897
100104
86.699
79 997
ASAM 1.0
82.27
91.913
86 504
87 896
77.526
99 422
85 884
87 631
64.521
761
84. 881
85 554
76.057
83,633
77 461
79.044
SAM 0.1
90.578
90.960
93.494
91.344
91.695
05.543
91. 508
92.915
88 58.664
646
99.14 19.163
98.15%
85.195
78.286
90.963
#4,814
We present ToW of Sharp MinMax and compare to Tab. 1. Compared with NegGrad and other meth- ods, Sharp MinMax further improves the unlearning capabilities across all settings by a noticeable margin, especially on Frigh. and SAM 0.1 achieves ToW > 0.9 for most settings on CIFAR- 100. The effectiveness of Sharp MinMax assures our assumptions about overfitting for unlearning, providing new insights for designing future unlearning algorithms. By constraining overfitting to only a small portion of model parameters which are most salient to F, Sharp MinMax effectively boosts unlearning performance. In App. G.5, the impact of relearning attacks Sharp MinMax is effectively limited to the sharp terrains as it makes retain and forget models geometrically distinct, so the robustness against relearning attacks as well as the model performance is retained.
4.3 QUANTITATIVE ANALYSIS AND VISUALIZZATONN
Measuring entanglement. We measure the entanglement between R and F before and after un- learning. At coarse level, we implement variance- based entanglement from Goldblum et al. (2020); Zhao et al. (2024): EAll F, = ( Ed -2R)²*++++++ !F E (())) (ML 1,) where di, Φj denote sample embedding, MR, fa denote mean embedding of R.F, and μr denotes mean embedding over R U J.F We also compute the class-wise entanglement and report weighted averaged E However. Evar assumes good/convex shapes of clusters and relies heavily on cluster means. Inspired by Optimal Transport literature, propose refined geometry-
8

---

Published as a conference paper at ICLR 2026
All Classes
Class 95
Clesses
Glss 93
Pelsh Farge
Kaze Tege:
Notan Parget
Tarpe
SAM pretrained, FAIl 61.74, FiB
49.88
SGD pretrained, FAIl
66.3, Fil
57.11
All Claoses
Class
Classss
Class
Reain
SAM NegGrad, B.
52.36, EW, 44.71
SGD NegGrad, EAll
52.99, EW,
46.91
All Clasess
cless 93
Dlsses
cless
Fage
Togat
Tore
Fagel
SAM MinMax, EW, 51.8, EW, = 44.56
SGD MinMax, EW, 53.7, EW, = 49.57
Figure 1: UMAP (McInnes et al., 2018) feature analysis on Mid Mem Fmid- At all-class level, we observe that SAM better maintains class clusters after unlearning while SGD is forming a more evident clump of features; at classwise level, we observe that while both push away forget features, SGD also scatters retain features further, suggesting overfitting. This also explains the larger clump of SGD at all-class level. We observe that SAM further pushes away forget features on Frigh and SGD scatters more retain features on Flow, see App. H.2 for full visualizations.
Table 4: Entanglement J between F and R of different memorization levels given models based on SGD and ASAM 1.0. While Fyar is hard conclude comparison between SGD and SAM across different 2, SAM shows less entanglement both before and after unlearning than SGD by Ew,
SGD
Variance Ex,
Wasserstein Eyr.
SAM
Variance Ev.
Wassersiein Eqr.
Modal
High
Mid
AVG
High
AVG
Model
High
Mid
AVG
High
AVG
Premmined
30.5
95.28
32.39
52.72
59.58
66.3
63.13
63.0
Pietrsined
29.56
85.43
25.91
48.97
55.86
61.74
59.84
59.15
2.5
6.71
3.91
51.21
class
2.88
6.66
4.08
45.45
49.26
NegGraal
18.87
37.16
51.24
$3.45
NegOrad
17.78
26
52.39
class
0.56
per class
0.66
36.42
MinMax
17.7
38.03
21.51
25.75
51.12
53.7
56.77
53.86
MinMax
16.35
32.07
20.75
23,06
51.25
51.8
55.08
52.71
-per class
0.69
2.41
2.27
1.99
38.41
49.57
57.15
48.38
-per class
0.49
1.52
2.97
1.66
33.65
44.56
52.55
43.59
aware entanglement based on Wasserstein distance to measure the separation of retain and forget features, EW,, and ETi,, which computes the cost transferring one shaped distribution to another wisnt-wisely. From Tab. 4, we observe that both SGD and SAM unlearning have decreased entan- glement with clss < All While Evar cannot further differentiate, we observe that SAM achieves better Ew, than SGD at all levels. Fig. visualizes the feature space of A, = ASAM 1.0 and A, = SGD on Fmid- For all classes, we observe forget samples are assigned to wrong class clus- ters after unlearning, where SAM better maintains class clusters. For class-wirs wise, visualize the largest class Fmni and observe that SGD unlearning scatters more retain samples than its SAM counterpart, suggesting overfitting. See App. H.2 for complete visualizations.

---

Published as a conference paper at ICLR 2026
Averae Across Nemortation Levels
Nem 5
Mem
Mem
34-
10
0.-
3.5
A.50 D:560
SGO 1-500
A.500.2500
A:500.1:500
S.M
A.SAM
A:5M 955AM
0.2
4500,01500
4550,
2 die 2M
Figure 2: As a decreases, NegGrad puts less weight on retain signals and learns more from F, leading to harmful overfitting. SAM exhibits more tolerance to insufficient retain signals, while A = SGD collapses the fastest. Note that ToW starts failing before a = |R|/([F + (R]), implying more factors affecting a threshold as we point out.
Basin: 50.24
Basin: 46 24
Basin: 40.8 80
Basin: 50.40
Basin: 28.16
Basin: 29, 29.60
Diest A = SAM
Diest, NG
Dies, MinMax
Fmidd A = SAM
Fmid, NG
Fmid. MinMax
Basin: 40.32
Basin: 42.56
Basin: 38.88
Basin: 43.52
Basin: 26.24
Basin: 23.36
Dua, A = SGD
Dies, NG
Dies, MinMax
Fmid, A = SGD
Fuid, NG
Fmid, MinMax
Figure 3: Loss landscapes on Diest and Fmid. where first row shows a SAM pretrained model and SAM unlearned models, and second row shows SGD counterparts. While unlearning increases sharpness as suggested by reduced basin ratios, we observe SAM unlearned models still maintain flatter landscapes than SGD models do.
Reducing retain signal. We verify Lemma 3.4 by reducing a in NegGrad. Fig. 2 shows ToW changes as a decreases for various A, U pairs at different memorization levels on CIFAR-100. We observe that A,4 = SGD fails the fastest and hardest, while A, = ASAM 1.0 exhibits the best resilience. Also note that for CIFAR- higher a. This supports our claim that a depends more than retain- ettain-forget ratio.
[ + [R]) = 0.93, but unlearning starts to fail at a
Loss landscape. We visualize loss landscapes of SGD and ASAM 1.0 by perturbing original model along two directions with filter normalization (Li et al., 2018), and quantify more sharp- ness by smaller basin ratio. Fig. shows loss landscapes on Diest and Fmid. where SAM unlearning generally keeps flatter landscapes. Same observations apply to different Fexcept that observe SGD+NegGrad on Fhigh to achieve flatter landscape, which might indicate that unlearning can be an implicit regularizer, we will leave it to future work. See full visualizations and more details in App. H.
5 CONCLUSION
In this paper. we provide a refined characterization of SAM under NegGrad unlearning, and the- oretical insights on bounding and choosing the weight factor to balance retain and forget signals. Extensive studies verify our analysis and reveals more underlying properties of SAM that are de- sired for unlearning. Based on our rethinking of overlitting, we also propose new algorithm which further pushes the boundary of sample-specifie unlearning. Our theoretical and empirical findings shed light on future design of unlearning algorithms.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Idan Attias, Gintare Karolina Dziugaite, Mahdi Haghifam, Roi Livni, and Daniel M Roy. Informa- tion complexity of stochastic convex optimization: Applications to generalization and memoriza- tion. arXiv preprint arXiv:2402.09337, 2024.
Peter 1. Bartlett, Philip M Long, and Olivier Bousquet. The dynamics of sharpness-awan minimization: Bouncing across ravines and drifting towards wide minima. arXiv preprint arXiv:2210.01513. 2022.
Marlon Becker, Frederick Altrock, and Benjamin Risse. Momentum-sam: Sharpness aware mini- mization without computational overhead. arXiv preprint arXiv:2401. /2033, 2024.
Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivan- shu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 16:28072-28008 2023.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv: 13427, 2019.
Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsieh, and Quanquan Gu. Why does charpness-awand aware minimization generalize better than sgd? Advances in neural information processing systems, 36:72325-72376, 2023.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning. arXiv preprint arXiv:2401. 2024.
Lue Devroye, Abbas Mehrabian, and Tommy Reddad. The total variation distance between high- dimensional gaussians with the same mean. arXiv preprint arXiv:1810008991, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-base based weight saliency in both image classification and generation. arXiv preprint arXiv: 23 /2508, 2023.
Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst- case forget sets in machine unlearning. European Conference on Computer Vision, pp. 278- 297. Springer, 2024.
Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards IIm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05374, 2025.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010001412, 2020.
Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. Attribute-to-deltte:: Machine unlearning via datamodel matching. arXiv preprint 1r.ii:2410.233..2 2024.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9304-9312, 2020.

---

Published as a conference paper at ICLR 2026
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriaa Cherepanova, and Tom Goldstein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In Interna- tional Conference on Machine Learning, pp. 3607-3616. PMLR, 2020.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 11516-11524, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: /903. 2019.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data dele- tion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016. PMLR, 2021.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584-511 16055, 2023.
Pham Khanh, Hoang-Chai Luong, Boris Mordukhovich, and Dat Tran. Fundamental convergence analysis of sharpness-ave aware minimization. Advances in Neural Information Processing Systems, 37:13149-13181 2024.
Young In Kim, Pratiksha Agrawal, Johannes O Royset, and Rajiv Khanna. On memorization and privacy risks of sharpness aware minimization. arXiv preprint /0.00483 2023.
Myeongseob Ko, Henry Li, Zhun Wang, Jonathan Patsenker, Jiachen Tianhao Wang, Qinbin Li, Ming Jin, Dawn Song, and Ruoxi Jia. Boosting alignment for post-unlearning text-to-image 2024.
generative models. Advances in Neural Information Processing Systems,
Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International conference on machine learning, pp. 17615- 17659, PMLR, 2023.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. https://www.cs.orrott.ooo eddi (ttp://mmmnnn-fftuurr-00000000-0
2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems, 36: 1987-9777 2023.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-
aware minimization for scale-invariant learning of deep neural networks. In International Con- ference on Machine Learning, pp. 5905-5914. PMLR, 2021.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Hao Li, Di Huang, Ziyu Wang, and Amir M Rahmani. Skewed memorization in large language models: Quantification and decomposition. arXiv preprint arXiv:2502.01 187, 2025.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-awant minimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 12360-12370, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05100 2017.
Joannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi, and Volkan Cevher. Ascent fails to forget. arXiv preprint arXiv:2509.26422 2025.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 03226, 2018.
12

---

Published as a conference paper at ICLR 2026
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make aware minimization stronger: A sparsified perturbation approach. Advances in Neural Information Processing Systems, 00.300000309623 2022.
sharpness-awan aware
USVSN Sai Prashanth, Alvin Deng, Kyle O'Brien, Jyothir SV, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, et al. Recite, reconstruct, recollect: Memorization in Ims as multifaceted phenomenon. arXiv preprint arXiv:2406.17746, 2024.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2535 2015.
Yan Scholten, Stephan Günnemann, and Leo Schwinn. A probabilisstic perspective unlearning and alignment for large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /oppnrpeeeieeweeeeeee//e/.e..eee net forummirdds -1i--11-11100 51WriiikK.....
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Pro- cessing Systems, 34: 1807-- 2021.
Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we mak- ing progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint 07X1::2406.09073. 2024.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Sheng- Yu Wang, Aaron Hertzmann, Alexei Efros, Lun-Yan Zhu, and Richard Zhang. Data attribution for text-to-image models by unlearning synthesized images. Advances in Neural Information Processing Systems, 37:4235-4266. 2024.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2 /577, 2021.
Lei Wu, Zhanxing Zhu, et al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arXiv:1706. 10239, 2017.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Advances in Neural Information Processing Systems, 37:36748 77736748-36776, 2024.
Zhe Zhang and Guanghui Lan. Solving convex smooth function constrained optimization is almost as easy as unconstrained optimization. arXiv preprint arXiv:2210.05807, 2022.
Kairan Zhao, Meghdad Kurmanji, George-Octaving Bärbulesou, Eleni Triantafillou, and Peter Tri- antafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37:12293-12333 2024.
13

---

Published as a conference paper at ICLR 2026
A Related Works
15
A.1 Machine Unlearning
15
A.2 Sharpness Aware Minimization
15
A.3 Data Memorization
15
B
Statements
16
B.1 Reproducibility Statement
16
B.2 LLM Usage Statement
16
B.3 Limitations and Future Work
16
Table of Notations
17
D Detailed Formulations and Proofs
18
D.1 Proof to Theorem 3.2
19
D.2 Proof to Theorem 3.3
25
D.2.1 Proof Lemma 3.1
25
D.3 Proof to Corollary 3.3.1
29
D.4 Proof to Lemma 3,4
30
E
Implementation Details
30
E.1
Experiment Setup
30
E.2 Sharp MinMax Implementation.
31
E.3 Unlearning Setup for Previous Work
E,4
Evaluation Details
32
Detailed Empirical Results
33
F.1
Statistical Significance
33
F2
Complete Accuracies
33
Additional Experiments
33
G.1
CIFAR-10
33
G.2
ImageNet
35
................................
G.3 Unlearning with Structured Noise
36
G.4 SAM with Adam and ViT
36
G.5 Relearning Attacks
37
G.6 Runtime and Efficient SAMs
37
G.7 KLoM Scores
37
H Complete Visualizations
39
H.1 Loss Landscape
39
H.2 Feature Visualization
39
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
A. MACHINE UNLEARNING
wide variety of unlearning algorithms have been proposed to erase the influence of specific data in the pre-trained model. Basic approaches involve finetuning retain set to unlearn the forget sam- ples with catastrophic forgetting, randomly labeling forget set to force the model to ignore the noisy forget samples, and explicitly "learning to unlearn" from the forget set via gradient ascent (Golatkar et al., 2020; Graves et al., 2021; Warnecke et al., 2021). Recent work pushes the boundaries of each genre with more advanced tools. LI-Sparse (Jia et al., 2023) finetunes on retain with L1 penalty to improve unlearning with sparsification, NegGrad and SCRUB (Kurmanji et al., 2023) combines gradient descent on retain set and gradient ascent on forget set to jointly update the model, Influence Unlearming and Saliency Unlearming (Izzo et al., 2021; Fan et al., 2023) aim to find model parameters which are important to the forget set for more effective unlearning while preserving model perfor- mance. Theoretical work in unlearning draws insights from differential privacy and characterizes distributional closeness in (c, d-language Sekhari et al. (2021) studies unlearning with second- order update which computes Hessian inverse. Langevin Unlearning (Chien et al., 2024) studies approximate unlearning with privacy and efficiency guarantees based on projected noisy gradient descent. Unlearning also extends to generative vision and language tasks, addressing privacy and safety concerns, erasing concepts, and aligning with human preference (Ko et al., 2024; Wang et al., 2024: Zhang et al., 2024; Scholten et al., 2025).
A.2 SHARPNESS AWARE MINIMIZATION
Sharpness-aware minimization (SAM) perturbs the model within a ball neighborhood maximize the loss. Since perturbations in sharp regions result in higher penalties, SAM learns to avoid sharp landscapes and improve generalization with flatness. Recent work improves SAM's flexibility and efficiency. Adaptive SAM (Kwon et al., 2021) introduces scale-invariant adaptive sharpness to ad- dress parameter re-scaling sensitivity. GA-SAM (Zhang & Lan, 2022) adapts the perturbation based on gradient strength to improve generalization performance. Sparse SAM (Mi et al., 2022) shows that adding sparsity in perturbations can preserve or even improve performance while accelerating training. LookSAM (Liu al., 2022) efficiently scales up SAM by only periodically computing the inner gradient ascent. Theoretical studies of SAM focus both on the convergence analysis (Khanh et al., 2024) and its dynamics (Bartlett al., 2022). Chen et al. (2023) reveal the fundamental mechanism of SAM that prevents memorizing noisy signals by deactivating neurons based on a practical signal-to-non noiise analytical framework. This inspires us to investigate the intriguing proper- ties of SAM in machine unlearning, where signals from the forget set can be naturally modeled as the noise from the perspective of maintaining model performance with remaining samples.
A.3 DATA MEMORIZATION
Recent work aims to identify key factors that affect the difficulty of an unlearning task. Fan et al. (2024) define and seek the "worst-case" forget set using gradient-based adversarial approach. Car- lini et al. (2019) investigates and quantifies the atypical-ness of data samples under a differential privacy setting. Zhao al. (2024) discovers that the more memorized the forget examples are, the harder unlearning becomes. We agree with the empirical studies in Zhao et al. (2024) and study the unlearning effectiveness under different levels of data memorization. Memorization literature provides fundamental understanding and interpretation of learning dynamics and model behaviors, characterizing generalization bounds and the interplay with data (Feldman & Zhang, 2020; Attias et al., 2024). Recent studies also investigate the effects of memorization in large-scale scenarios such as language models (Biderman et al., 2023; Prashanth et al., 2024; Li et al., 2025). Specifi- cally, the memorization and influence scores in Feldman (2020); Feldman & Zhang (2020) provide insights into evaluating unlearning algorithms and designing new approaches. In our study, we have observed varied effectiveness of each unlearinn method with respect to forget sets of different memorization levels, and aim at designing unlearning methods which perform well on forgets sets of all difficulties.
15

---

Published as a conference paper at ICLR 2026
B STATEMENTS
B.I REPRODUCIBILITY STATEMENT
Experiment environment. Our code is built upon several open-source code bases and will be released. We perform all experiments on single NVIDIA A100/H100. We fix random seed for all data processing, saved precomputation (e.g., indices for data subsetting, weight masks), model splitting, pretraining and retraining for reproducible observations. For unlearning parameters and settings, we run experiments with multiple seeds to evaluate statistical significance, App. F.I.
Theoretical Assumptions. Our theoretical analysis follows standard, existing assumptions of model size, data size, effective information in the data (signal) and Gaussian noise in data, which were previously stated in Kou et al. (2023); Chen et al. (2023). In addition to mentioned common as- sumptions, our Assumption D.1 also assumes conventional unlearning schemes: cross-entropy loss, ReLU activation, clean labels and reasonable size of forget set (< 1/2 trainset size).
B.2 LLM USAGE STATEMENT
We use GPT to fix grammar and polish short phrases to sharpen our expression. We also use GPT as a smart search engine to gather recent work of interest and summarize existing bug fixes. Zero LLM usage for any core component of our work, including data processing, implementation and experiment, theory, etc., and LLM does not guide the development of any module. No "vibe coding" and mathematical derivation from LLM.
B.3 LIMITATIONS AND FUTURE WORK
There are a few limitations based the signal-to-noise framework, which on the other hand inspire us for future studies. First, there are more interference which can be modeled as noise in machine unlearning, such as the overlap between retain set and forget set. Using hard-cutoff or random sam- pling to build F might split two similar samples into two opposite subsets, causing interference and impacting unlearning effécctivneess. We hypothesize that less overlap between R. and JF results in more effective unlearning, and vice versa. With more identified and modeled noise sources, another limitation comes from the uncharacterized behaviors when retain signals are weak for some upper bound. Will SAM fail into harmful overfitting under this cireumstance? Theoretical and empiri- cal studies under this situation might leverage the interplay between all signals, including different noisy signals. Another limitation comes from the design of ascent-based unlearning like NegGrad as discussed by a concurrent work Mavrothalassitis et al. (2025): while the objective encourages misclassification, the targeted retrained model should treat forget samples ing or guessing). This misaligned objective might cause potential imprecision and leakage. While show that SAM's benefits trivially apply randomness- based unlearning (e.g. RL in Eqn. 3), and ascent-basee unlearning is widely adopted in frontiers (LLM, diffusions), deeper studies are expected for more robust unlearning. Last, we observe an intriguing "regularizing" effect of un- learning using SGD via loss landscape visualization, which demands deeper investigation in future work.
seen (generaliz-
hhttps://githon http https: waaelllannn-aannneee http:://ploakkd..githhtt
to/influence-ment zationn
16

---

Published as a conference paper at ICLR 2026
C
TABLE OF NOTATIONS
Table 5:
Symbol
Meaning / Notes
Symbol
Meaning, Notes
X; E D2 Pxd
Input image of sample i, vectorized into P patches of dimension d (one patch holds the signal y and P-1 patches contain noise)
ys E {+1}
Binary class label for sample
$ IRd
Universal signal vector shared across samples
B
Noise vector for sample often drawn from (O, o
P
Number of patches per input image
d
Dimensionality of each patch and
each convolutional filter
Number of convolutional filters
Wirr R
Weight vector for the r-th filter of class j [+1]
class
W
Collection of filters {w5.r)m rim for class j
w
Complete set model parameters
f(W,x)
Two-class CNN output: f+1 (W+,,x) - (W1,))
f (W,,))
Class-j output:
- Epur ((W5)r;)p)))
ar-)
ReLU activation function
d'(-)
Derivative of ReLU used in gradients
((W,S)
Cross-entropy loss over training
Gradient of the loss for sample epoch f, batch b
(C.8)
r-th filter of class j E {=1} after epochs and batches
(0,b)
Learned signal coefficient for filter (j,r) at step (t, 6)
Sina
Learned noise coefficient from sample filter (j,r) step (f,
||<e| la, ||e:/22
Euclidean norms of the signal and noise vectors
FCS
Forget set whose influence is to be removed
R = S S\F
Retain set used for continued training
A
Model after Ti epochs of training algorithm A
M
Model after T2 epochs of unlearning by algorithm Ut
Ti.Th
Numbers of epochs for pretraining and unlearning
I IT
Mini-batch indices from R and F. step (t,
Batch size
Learning rate
sgn(.)
Sign function returning 1
Weight in NegGrad balancing retain and forget contributions
(t..6) J.r
SAM perturbation applied to Jar
(6.6)
T.P
Perturbation radius in theory and in practice used in SAM/ASAM
8
Perturbation term: 5 = ((..)) SAM and 0 for SGD 5.r for
PP V
Gradient contributions for the signal and noise in NegGrad updates
SAM A spoch jur
Per-epoch change of Kg.r under SAM
SGD Spoch Kijor
Per-epoch change of Kg,r under SGD
Acc(0, D)
Classification accuracy model on dataset; 0, D are abhreviated terms in Acc()
ToW(Ju)
"Tug-of-war" metric combining retain, forget and test accuracies
D
data distribution
Frigh
Forget sets of high memorization difficulty; same for mid, low
(,i,
Memorization score: Pr(J(S)= = v.] Pr[ \ v.]
Training set S with sample removed
Feature embedding of sample i used entanglement analysis
Mean embeddings of retain set, forget and all data
Continued on next page
17

---

Published as a conference paper at ICLR 2026
Table 5: (Continued)
Symbol
Meaning / Notes
Symbol
Meaning / Notes
E (R, F)
Variance- entanglement measure between R and F. given model f
ECs
Class glass-wisc version of the variance-based entanglement
W.p BWP
Geometry-awee aware entanglement measures based Wasserstein distance (all/class-wise)
A, u
Training algorithm (e.g. SGD, SAM) and unlearning algorithm (e.g. NegGrad, RI., with default SGD optimization, can be used with SAM)
(0.0)
Initial signal coefficient for filter
(0.0) Sjan
Initial noise coefficient for sample on filter (j.r) Total number of samples ((S))
(j.))
[F], [R]
Cardinalities of the forget and retain
sets, which is size in our work
Diest
Test dataset used for evaluation
SGD
a SAM
weight coeff for SGD and SAM, respectively
D DETAILED FORMULATIONS AND PROOFS
We prove our theorems and lemmas based on previous theoretical results in Kou et al. (2023); Chen et al. (2023). Specifically, we prove that with additional yet necessary conditions for effective un- learning, the final test errоrs be preserved, while we identify and characterize the changed inter- nal dynamics. We begin by expanding and restating Hs Ç update rule for NegGrad in Eq. 7:
(1,b+1) (t.b) ji 1.r
((,bb) o d(w (1,b) + A. A.y))))
Bm
EEZ
((tbb) O(ww (e,b) A v(sp))
-(1 a1
TEI
(1,b+1) (11.8) -((1b) 1/P- - 1)2
| (( d'((
(t.b)
+ +.)))) 1(/i = =j)
frr jjr
Bm
jur
VETYY
-(1 - a) Σ
d'(w (1.b) (
+ +, 1( = =)
+ T/P 1)2
(1.+1)
((1.b)
| ((ttt o' d'( + +. 13 +
(t.0)
Sjr
Bm
VEIR
-11
Σ 1E d'((w + +, 1(v: +
(1.b)
(11)
where A ((f.b) for SAM and 0 for SGD, Go is split into Gjr
(t,b) = f/r (c (t.b)
> 0) and
(t.B) = 1( f.r Sor
((.b) < 0) based on label agreement. We summarize several reasonable assump-
tions from previous work in addition to our conditions which ensure unlearning to progress:
Assumption D.1 Suppose there exists sufficiently large constant C, such that the following hold:
1. Sufficiently large dimension d: d > C max 2| log(T"*), n²² (og(m)m/))(oog(())))) for some T -2n 1Bmd-111 1p-2--p-22--- 2).
18

---

Published as a conference paper at ICLR 2026
2. The size of S and the CNN width satisfy n 2 C'log(m/6), C'log(n/0).
3. The signal strength satisfies 1| 2 Co2 log(n/6).
4. For the Gaussian noise initialization, do S (C maxffxdd Vn, Vlog(m/o) 1|
5. The learning rate 1i satisfies dl - (C max (n0m:///(nmmmmg///)))o)... o3dd///))))
6. Assume cross-entropy loss: {(z) log(1 1 + exp(-
= -110
7. Assume ReLU activation.
8. Assume all clean labels and F signals do not dominate: 2 |R| /(1]+ (R]) 3 >> 0.5.
We then obtain several proven quantities from previous work, which are achieved during pretraining and can be leveraged at the start of unlearning:
Ex= T S.. @(SNR "), for the signal-to-noiss ratio SNR
14212 (P (P-1)o Vi
E Sjrri = ((())= = On log = (()) for some T = 12(11 Bmdd 1 p-207²).
maxj,n, ( = max (000loolmmm/)) adop O( /log(n/0)log(T)) n/ n/Vd]
(T (R), where = n. S
D.1 PROOF TO THEOREM 3.2
Under NegGrad, we want to predict retain samples in R correctly while we count correct predictions in F as errors, yielding same bounds for (x.)))< (yf(W()) x) S 0) and IP (xxx)>>((ff (W)(( ,x) > 0) based on inverse objectives. However, when considering the test error on the model that is jointly updated by gradient descent on R and gradient ascent on F, we still measure the error rate by wrong predictions. In other words, fitting forget samples will reduce the generalization performance. We can decompose the test eтror as follows:
PP ((xx))> y # sign (s (w((,x))) = =P(x (ns (wio), ,x) $0)
=Ip (xxx)~D (ff (Wr') ,x) < (x, 3) R + P Pxxxx)) (vf (wr()) xx) 50 (x, y) FF)
(12)
=3 -B3PP(x)) ((xx))< yf (w()) x
s 0 + +11-3) 9)) P((x)))=F yff (W()),x 50)
=8 - ((x))R uf ((()) x 50)= + +11- 9) (1- (vf W(O)
Note that in practice, R and F come from training set S. During inference and evaluation, we convert the data augmentations of R, to test transforms, thus measuring proxy-test errors on R- like and F-like samples. To bound the test error, first decompose yf(W (W(()) x) into signal and noise learning of both positive and negative classes, considering A = 0 for SGD:
ws (w())xx x
(wwC)), 1 Eui ui [o (( J.r. (1) + ((w.10 5) $
j,r
1
[o ((wit)) y wpy) ) + +(P (P-1)0 ((m()))))
(13)
m
m Σ
I (r. + (P- - 1)0 (ww()...)))
Remark D.2 The following proof process for bounding P( (anhwreyf(Wi (W()) x) comes from Kou et al. (2023). We include it here for readability, since we will leverage the results when combining R and F in the end, as well make adaptations for proving Theorem 3.3. Our results benefit from previous work as we consider the unlearning process as an extension of the second stage in Chen al. (2023).
19

---

Published as a conference paper at ICLR 2026
We begin by two lemmas that bound the signal, noise norm, and the related inner products:
Lemma D.3 (Lemma B.4 in Kou al. (2023)). Suppose that å > 0 and d [2(log(nn/0) (6)) Then
with probability at least 1 - 5,
ofd/2 - ||6 30/d/2.
((), S 20 vd (6n 16),
()) s 11 /2log(6n/8)
for all i, E [m].
Lemma D.4 (Lemma B.5 in Kou et al. (2023)). (1)o)((/))) Then with probability at least 1 - 6,
Suppose that d ([llg(mnn00) m
odd/2 =
w jir
(0.0)
2 30dd/2.
2
(ww] Jir
(0.0)
.p)
S 2l (ogmmlmm/)) col|v|12,
(0.0) ((.0))) w J.r E s 2 2
ooo, vd.
for all r E [m], E (+1) and E [n]. Moreover,
aol1y|12/2
max rE(m)
(w000))) .p) S V2l2og /
0oop 0oo,/ vd/4 S max j rmmm
( (w).r & 12g(((1)))))) 000,
(0,0)
for all j E [=1} and i [ns]
Plug in the weight update decomposition in Eq. 2, we can first bound the inner product for j = y:
(1)
04)) )
(
(w(()) y.m: w46) > + xx(()
1
Mara 115:112 V4p)
UPP) + p
||
(E:)))))
P
i=1
2
/)) 00114|12 12 +n (1)
2 log (og(66/0)
P-1 op 1111 (0,d/2)
-1
Σ
yorsi
+
Synr
i=1
θ (Vlog(m/5)ooll (+ll)) + Nig - (Vlog(n/5) (Pozd) - 140112) θ (SNR-2) ((()) (Vlg(m/0) (a -1 val(0l12) + [1 11-6 0
(t)
θ
(Vlog(n/o)-Pogg Popp Popl/(<ll2) kig.
(1)
(8) =0 (ggr)
(14)
where the inequality is by Lemma D.3 and Lemma D.4; the second equality is obtained by plug- ging in the coefficient orders we summarized at the beginning of the section; the third equal- ity is by do I (a,dd /n in Assumption D.1 and SNR = ||<ll2/(((P - Vd), The SNR?. Also Vlog(n/8) op/lllple S I /lggmm/))/r 1/(C //o(//)) holds by ||gl C. og log(n/5) and n 2 C ( in Assumption D.I, so for sufficiently large constant C' the equality holds. Similarly, we can show that i Fy. y.
(t) fourth equality is by Ko O(R), where R
1/VC and ((og(m/)(a,,d)1) plll/// = /
(w (9) -yr y(p) = (k)))) < 0 for j
(1)
Next denote 9 as Σ, a(ww"( -g. E)). Since E ~ NO I),, we can leverage the Gaussian con- centration bound for 2 0:
cx2
P(g(E) Eg(E) 2 x) < exp (-
(15)
Lip
20

---

Published as a conference paper at ICLR 2026
where è is a constant. To calculate the Lipschitz norm, we have
l0(5)
9 (5') =
Σ
( ( v.r. {)) - -E
(ww)) -.. (.
r=1
r=1
m
Σ
0
w B.r 5)) -
(ww 55
(16)
r=1
m
m
(1) p.r
5
|| ||E 5'1l2- y0
The first inequality is by triangle inequality; the second inequality is by the property of ReLU; the last inequality is by Cauchy-Schwartz inequality. Therefore, we have | Itap S >m || and since (w y.r E) NO.. [[ y.r ((2o)))) we can get
ll..
Eg(E)
Σ Eo (w.).. 5))
m
m
p
Pp
m
w
(t)
(17)
2T
2T
v.r
2
Then, we seek to upper bound the 2-norm of w wa.r First we have
(t)
n
Σ Sh 1|
E
Sina
2
1|5i112 +2
Σ
Gonn Sirii 16,,12 21 112 - Eiz)
disgonal
off-diagonal
<40,
p
2d
2 +2
Σ
Sirsh Sjarée
(160 (207) / /dlo (6n
15 sin
(18)
_0 2d
(1) 2 32a 22 3/2
2/2
v log (6n2/8)
(4) 2
S.z
2
=0 (0 2 d1)
(1)
+ 1 0 22223112)
j.rt
S
1=1
12
n
$ (0 In + θ ( 2
Σ
Σ
S
T=1
2
0 (0 2n
The first inequality is by Lemma D.3; for the second inequality we used the definition of 7, <: for the second to last equation we plugged in coefficient orders. We can thus upper bound the 2-norm
of !!! J.r as:
(4)
<
w
(0)
(c)
||
2
+ p
1
1|
E.
<
w
j.r
(0)
(0)
|| + θ (
(p-loz d 1/2n -1/2)
Aind
(19)
0
(p-1 1 11 1/2 -122)
Σ jinp
20
1=1
21

---

Published as a conference paper at ICLR 2026
where the first inequality is due to the triangle inequality, and the equality is due to the following:
K 1|:0112
= =
( d- 1 1/2) Ei=1 7(1)
jjnnd
(20)
=O p p 1/2 1/2 1101112 = O(),,
based on the coefficient order Ei= ((0) / OSSNR ?), the definition of SNR, and the con- dition for d in Assumption D.I. Similarly,
(0) f.r
θ (oova) O O d d-1/2d1 11/2) Ei=1 Syza
2 1-1/2n 1/2) E.= 0 (P-100
(1) Sjrss
(21)
=0 Pooorddn 1/2) - = 0(1),
based on Lemma D.4, the coefficient order Σ Sj = M(n), and the condition for on in As- sumption D.I. Then we can give analysis of the following key component:
Σ. E
+
Er
Kigir
(P Er=
(t)
θ
(d-1//2 Er,' 5-
(22)
-yr 1
1/2 (n2222 = θ
(1122111131
Then for ||4||2 2 C2/4 n 1/4 Popd for some large constant C1, we have
Σo (w// (P 1)0p 11ppp
w (1) yor
O.
(23)
2T
Upper bound. Now plug in previous results to obtain
P P(x,))~R (uf (w(l),x) x) 50) S (xxx)))~ (P (P- E5((ww))
((w")) (1)
(w").rrt)
2 Σ ((w(i)
(mn))-ww)))
-x(xx)> (((E) - Eg(E) 2 1/(P (1) ((w5th wirs wpp
w
(1)
2TT
V.r
- (1(P-1)E,
(( (ww00 (wrr (19)))s)))
(1)
m (ap// /2T
-
(e)
<exp
oi Em
v.r
Σ, a ((( (w ((w w/p)
27
2T 2T
exp
(P 1)0p Em
w (t)
Σ,
(( ((w ".y.r:00 844))
S <exp(c/2T)) exp
-0.5c
m (P- 1)0p
Σm
w-g.r
(1)
(24)
22

---

Published as a conference paper at ICLR 2026
The second inequality is by Eq. 23 and plugging lloliap S Σ || l| -y.rll into Eq. 15; the third inequality is due to (s - t)² s² /2 t², Vs,t t 2 O. And from Eq. and Eq. 24 we have
E, a (( (wobe y4
P ((x.)) (ns (wll), .x) 50) & exp(c/2x) exp
-0.5c
(P' - 1)0p D
W-y.r
n2|10112
c
exp
2T
C(P
1)1o3d
n2|16112
exp
(C
1)
n2140113
exp
C2((P -2(
C,
(25)
where C (1); the last inequality holds if we choose C1 cC/T; the last equality holds if choose C2 as 2C.
For the forget set JF thus have
P P(x.))== (vf (wit), xx) >0) Sc.
(26)
Lower bound. Without loss of generality, let Σ, (1)
1,r max E, 11 E, (0) [_r)
E,
Denote
v == > E, 1 (y) = 1) & where À = CySNR² = CT11p11// ((P - 1) and C7 is sufficiently
large constant. Since ReLU is convex, we have
o (( ( + v)) (( ((w(.r k) 20 o ((w1)) 5)) (w(t. v): (t)
(27)
((wfi E +c++ -a ((w) ((w1)) -x)) 2 o (w - ( (w v)
Summing the above two, have that almost surely for all g
w W ++ -o ((w(l)))) + -E - -o -0((11) -t))
(
(1)
(28)
(t) Sinaa 2n logglo0mm/////// ooo, Vd 5n²a log (6n2 /d
where the last inequality is by Lemma C.3 in Kou et al. (2023) and Lemma D.4. Additionally, since ReLU is Liptchitz, also have that
0
w -1r + v)) ++
(t)
((w")..:)) +o ((w"0, - -(+v)) - ((w_9,r:-
2
(1)
1,r,
(29)
2
Σ (t)
Irr + 2n 22n log(1mm)/6)) 0oo, V 5n 5nta log (6n² (8)22/8)/
23

---

Published as a conference paper at ICLR 2026
Therefore, by plugging Eq. 28 and Eq. 29, we have that ( + v) 9(E) + - -(
Σ Si,r.
6nm / log log(Lmmm/01 0000 Vd 15mm aannaa/loooo (6n² / /d
2 (A/2)
(30)
91=1
2 2/2 θ (SNR 3)
Σ
2 4C
Σ
(0) 1,
where the second inequality is by Lemma D.1 in Kou et al. (2023) and Assumption D.1; the third inequality is by Ei=1 y' O(SNR 3). Finally, it worth noting that the norm
(
= nl|g>1 2 Paood
|v|la x Σ
1 (=== E)
S 0.060pp
(31)
where the last inequality by condition ||و||| Cgddh/nn11/44 with sufficiently large Cz. Then we present a Lemma which bounds the Total Variation (TV) distance between two Gaussian with the same covariance matrix.
Lemma D.5 (Proposition 2.1 by Devroye et al. (2018)). The TV distance between N (0, (0, and N TT) is smaller than 2/
Finally, we can prove the lower bound for R:
uf (W()), xx) s 20)
P(x.))~
Σo 0 (( ((w (T.r) 5)) Eo ((mm00 2 >0o
((w / ( wp))
- -Eo 20((m) ((w0) (w
=P -P(x))>
20.5IP (x.))RR
(Eo ((w(())
- E)) ((((
- -o ((w!l (i (mm))-))) Co max
Σ
(1)
ks
-1
(2)
(32)
large, we can always pick a corresponding y given & to make wrong prediction. 1,m+
where Co is constant, the inequality holds since if Σ, ((w {) E, ((
1.n,
(t)
{) ) is too
Let g(E) = Σ, ((w(() 1.r 2)) - E, a ((
(c) 1,r E)), and denote the set f = {E lo(E) 2
C6 max 1i,r Σ, 1 1. Thus we have
P P(x.))<R (uf (wr(1),, , xx) 20) 2 0.5..
(33)
By Lemma 5.8 of Kou et al. (2023), we have that E, + +v) -(- 2 4C max; { [E,
(t)
(t)
K,r
Therefore, by pigeonhole principle, one of [g. - v, - + v] must belong to Ω thus QU-QU
Ω [v} U -Ω {v} = _ Therefore, at least one of P(1)), P(2 {v)), PP (v})
greater than 1. Note that IP(- P(2) and
PP(2) P(Ω v)1 (e (EE P -- (E e)
s TV (N (0, of .N (v,o,I1))
(34)
1v/22
< 0.03.
20p
24

---

Published as a conference paper at ICLR 2026
where the first inequality is by the definition of TV distance, the second inequality is by Lemma D.5. Hence, we have that P( 2 0.03 = 0.22, and plugging this into Eq. 33, we get
P P(x.y)~<R (uf (w x
= 0) 2 0.5P( = 0.11 2 0.1.
(35)
Like the upper bound, the derived lower bounds also applies to (((x ((xx)))~ (yf x) > 0). Hence, if || 2 1oll//nn
( D) = (x(x))) (y # sign (f (WT2,x)))
=3.1P (x.))) (uf (W x) 0) +(1 - 8) [ (uf (W x) >0)
(36)
SeR
SeF
- lim C lim
(WoH D) < ER c.
On the other hand, when / - 0.5, we have lim L (W D) S 0.5 + 0.5eR - 0.5cy =
Depending on the size ratio of R and F, e ranges from a very small constant to a minimally PAC- learnable threshold.
For harmful overfitting where ||0||2 S Czd Pop
C (WT2 D) (x (y # sign (f
=3. (x..)) (yf (W x) S 0) +(1 - ()
(xP (us (W x) > 0)
(37)
20.1
20.1
= lim C (W To D) 2 0.1.
On the other hand, when 3 + 0.5, we have l.m (WTo, D) 2 0.05.
D.2 PROOF TO THEOREM 3.3
First we have the same decomposition for NegGrad:
L (WT2,
WW,,D) D) =P (x,))~D ( + sign (f (w .x)))
=B P. ((x,)) (n) (w(), x) s + (1- 33- (1 P Pixen (nl (wr)) ,x) > 00);
uf (wro), xx
Σ ツ 0 j.r j.n ) + (w/0 5 5))]
(w (t)
-
j.r
Σ ((
((()
(9/)) +(P - 1)0 ((wl))
(t)
( ((w()..))))) (P- - -1)0 ((w()rr)))
m
(38)
However, note that for (x,y) ~ F, SAM gives up its denoising property. We first show this by proving Lemma 3.1.
D.2.1 PROOF TO LEMMA 3.1
Proof. Consider extending Lemma D.5 in Chen al. (2023) to the NegGrad setting by rewriting
(c(l .E First we have the Frobenius norm upper bounded by the same quantity:
(w.C,,,((( W (W(L.))) ]] = VlaV (W((.))) (1 a:) V VwLr,, (W((.))) sallV CIR, (W(1.))) ||F +(1 aallV////. WWJ (wird)))
(39)
= =/V (Wit S 22/2Pop d/Bm,
25

---

Published as a conference paper at ICLR 2026
where the first inequality comes from triangle inequality; the second equality holds because R, F are split from S and come from the same D, thus having the same gradient norm; the second inequality
comes from the original bounds in Chen et al. (2023). Next we expand (c(tee) Ex)
under NegGrad:
(c// (..)),ss))))>)))
-
V (wCr.o(W(lo)) (Wil)))
Σ
Σ
e y:o y:o'((w) /0 (xx,)) Ex)
mB
iEI,. rE(P)
[ (Wittb))
Σ
2 yio alo xix)) (xi, Ex Ex)
mB
SEZEA PE(PI
SEZ
(1 0))
ΣΣ
e vio ((w'((W))) jir X (x,))) Ek)
VET PE[P)
(40)
Note that (xi.ps Ek) can be divided into three different terms:
1|Ex112 S 30
if = k. 2x = Ex
Kxt,p Ex)1
(E,, Ex)I S 20 s/dlog(6n2))
if i F Zi. E if Tj.p = 4:4
(41)
(yis.. Ex)l | ||11|2 op log (6n (6n²/8),
The upper bounds come from Lemma D.3. Based on Assumption D.I and Lemma D.4 of Chen et al. (2023), the i k term will dominate the upper bound and write
(Lb)) for .6
O.15a(PP 1)C1o T.,))
mB. 2/22o, d Bm
(42)
+0.15(1 +0.5(1-0)P-1))ioo001 [7,]]]
Thus, when & E IRR. can preserve the original bound with additional a:
((.b) for .Ex < aTOP Vd mVB B
(43)
Choosing
m
will cancel with ( (w (1)
( Ex
deactivate the neuron. When & E IT,66 the
Caa Va (t.6) (t.b)
entire (w
jr
jrr
Ex) will remain activated:
c <c(--a)ro,va a)
0 < (c(t))) Exx
<
(w(())) +
(1.b)
(1.b) Grr EEx
(t,b) for
Ex
2 0.
(44)
mvB B
This fundamentally differs SAM's behaviors towards unlearning F from behaviors towards learning R how SGD differs from SAM. For gradient ascent F under NegGrad, we know SAM learns from activated noise products as much as SGD. The activation patterns are further utilized to bound products and norms of the weight, signal and noise, which characterize the final test emrrs.
26

---

Published as a conference paper at ICLR 2026
Our task is reduced to bounding P (xxx)) Ruf (W()), .x) 0), then use previous error bounds for SGD in App. D.1 for P P(x.)=F(V(((() x) > 0). The inner product with j = y can be bounded as
(with w1p)
-(w (wi/0) (0)
y: w1p
+ 10
1
Synt 1|E:1122 (E:, y4p)
(P- 1)
+ P 1)
(t)
11::1122 14P)
v.r.
w (0)
y.rt W1P)
+ kighr
(2l0/(6n/0))
P 1
op 0pl1y1122 (o,d/2)
Syrrs +
(())
(())
Syrrel
1=1
(0) Nir w4p)
+ x 0 (Vlog(n/8) (Ppp) -1 1|<011))
@ (SNR 2) n.(()
+ [1 θ (Vloolu/60--Po//llllll night
yirr w&
(w)))) Mirt w4p)
+
(2) v. ((1,
(45)
where the inequality is by Lemma D.3; the second equality is obtained by plugging in the coefficient orders we summarized; the third equality is by SNR ||//ll/ (Ppp(PPp)))) vdl; the fourth equality is by
11 2 C. oo 1og(n/6) in Assumption D.1 for sufficiently large constant C; the last equality is by Lemma D.7 of Chen et al. (2023). We similarly have (w..r. v()) (1) < O.
Denote g(E) as Σ, ((w (1)
mir 3) The results for noise learning from SGD in App. D.1 still apply: -Mir
I0(E) - g (5') $
w
||E - E'1l2:
y:r
Jp m
Eg(E)
(t)
(46)
2
( Sind
||E: &:
<
0 (ap 2d'nn n 1)
Σ
(1)
Sprst
We can thus upper bound the 2-norm of w as:
(1)
(t)
ww(0
(0)
11
+ P-1 Σ
1
a
||E:lla 2.
E
jor
2
<
(0)
(0)
|1 + +
p -d 1/2 1/2)
Sind
(t)
(47)
= O(oov 0) 10 1/2 1122)
((1) jint
based on SNR /| (Po, vd) and Σmm
(11) / j.r e (SNR (ANN-2),, and the condition for d in
Assumption D. 1, and also w (0)
j.r = θ (oova) d based on Lemma D.7 of Chen et al. (2023), Then
27

---

Published as a conference paper at ICLR 2026
have
Σ, (wwhr
( (Wiir: yPp)
(
(P- - (1) -x=1
w (t)
(oovd) + 0 1 d Ei=
Sjrrs
(0)
W-BT 2
O(1)
[ov +0 (p-1op' 1pp20//22111///ppp
(48)
min
st 0 -o'od 122 1!
21
- =
(P
1)0p
m
(t)
w
-,, 2
> 0.
v 2m
Upper bound. Now plug in previous results to obtain
P(x,))R (us (wit), (x((),,) x) 50) < ((.,))<
(P (P-1)
((w!)...))
2 Σ
((w sirs vsss)) ((w)))-w))))
((
9(E) =P(xa)~R ((E) - Eg(E) 2 1/(P 1)
2,1r ws²)
Pp
Σ
ww -y. 112
2m
(1/(P- - [1((P-1),+ ((w (w wup)
Wgrr
(ap/ 22T
m
w
(1)
27
Sexp
of Σm
(t)
Mor
Σn 0 (( (w M.r: wyy)
27
= cxp
-c
1/V2T
(P 1)0, Emm
w -B.r
(6)
Σ,
((
(ww Wyrrs y14))
<exp(c/27)exp exp
-0.5c
(P- - 1)0p Em
w --yr
(49)
The second inequality is by Eq. 48 and plugging IIa|اu S Em l| - lll into Eq. 15, the third inequality is because (s - t)2 z /2/2 - 22, Vs,t 20. O. And we can obtain
P P(x))~R (ns (wit),,x 20) < (oxp(c/2x)ene -0.5c Σ, (( (w.yr 844
(P 1)0, Em=
(!) --i1
- mmin 2d- Poždn -tan--aa00 (a=
exp
2T
< exp (-0.5C min (oo 2d 2d Poÿ -1a -2]) E,
(50)
where = 0(1), the last inequality holds since of < 0.5Cd log(1/e) and d > 1C-1p-1o77 op ? log(1/c). Now upper bound the test error L D). Depending the strength of the unified signal vector sp the unlearning of F can exhibit either benign or harmful overfitting following SGD's characterization, dividing error bounds into two cases:
28

---

Published as a conference paper at ICLR 2026
1. If 11pl122 2
we have benign overfitting both R and F. Thus,
h [ D) (x.x)> (u # sign (ʃ (W??,x) xx))
=8 P (x,U) (ys (WTs, x) 00) +(1 - 8) (1-((x))me (us (WTs, x) > 0)
SER
Sef
lim 0-1
L
( ( CR
(51)
As 3 + 1. Fl/n decreases so the model can better maintain its performance; as - 0.5, [Fl/n increases and more samples are to be unlearne,, making the model per- formance reduce minimally PAC-learnable guarantee. Hence, when 3 - 0.5, we have
lim L (W D) S 0.5 + 0.5cR 0.5e =
2. (1) < ||o||2 S Cid 1// 11/Poo we have benign overfitting on R. and harmful over- fitting on F. Thus,
Lee (W D) P P(x,))DD (u # sign (s ((wl))))))
=8 P (x.g)<R yf (wl))xx x 50) +(1 -11-9)- 1- Px (uf (wro), x =0)
Sex
20.1
lim Leodt (WT2 S ER =. 8+1
(52)
Similarly, we have lim L (WT: D) < 0.5cR + 0.45 = 6.
Remark D.6 (3-dependence of the e-bound). The overall test error
Crest (WTx T2 D) = (x.)) (vf (W(),,
(uf (wi)) x)) > o))
can be considered as an affine function of the mixing factor B, and so its achievable range runs from the best-case retain error ER (as 3 + 1) up asymptotically 0.5 (as 3 > 0.5) trivial -eCalbaanability threshold. Concretely, by choosing 8 sufficiently close to 1, one drives grest (WT2, .D) arbitrarity close the small "benign" error level c, whereas if 3 remains near 0.5 then L (W D) can approach 0.5, the worst-case "minimally learnable" error. Thus, all our bounds interpolate smoothly between these two extremes via the single parameter B, and we report the most informative bounds in Theorem 3.2 and Theorem 3.3.
D.3 PROOF TO COROLLARY 3.3.1
Recall the update rule for Kj,r- For each epoch, the interference between retain and forget signals can be measured as
RI/BB Σa (uui) ((w! NEIR
FI/B
(1.) ((p)
Σ
(1-a)
R F
(1.b))
(1,b) o' d((w!rr .yup)).
(53)
iEI
Similar to Lemma 3.1, the expected gradient values between retain and forget samples should not differ. Since cycle the forget set to synchronously train with the retain updates from F has been scaled up by [FT Hence,
[R]
R
FF/B
Σ ((b o' ((w ((sp))
(1,b)
=E E
Σ
Σ
(t.b) o
d'(w
(t,b) jrr ((p))
(54)
29

---

Published as a conference paper at ICLR 2026
Combining together, to expect Kj,r increase monotonically every epoch, we want
RI/R
B
FI/B
E
a
(t,b)
o' ((W
(f.b) jor ((()))
Σ
(1-a)
F
R
((t.b)
d'((w (1.b)
wisp)
20
b
b
TE
R
R
F
+ R
(55)
D.4 PROOF TO LEMMA 3.4
By Theorem 3.3, SAM turns off noise memorization prevention mechanism when fitting F. which leads to the same requirement on signal strength SGD. The only difference between SAM and SGD under NegGrad is the more effective learning on R. From Eq. 7 we have the per-batch update of Kij.r on R as
n||4ا13 a
ANj.r
Σ ( ((w .v()))
Bm
(56)
iET,
Let denote the batch-average magnitude of C((-B)) express per-epoch update as
d'(
(f,6) (k()) for convenience. We can then (f. J.r
0||6| R
pppch
(57)
m
Now, consider achieving benign overfitting on R only, where SGD requires ||و||2 = (dd (R| Pop) while SAM only requires ||ş|||2 = 2(1). That being said, given fixed univer- sal yp for ID and a choice of a, we have SAM learning the retain signals faster than SGD:
SAM eepoch
SGD = 0(d'/2 1/2 1222)) =
@(1|p 12).
(58)
Aepuch
Hence, in order to achieve the same signal learning performance as SAM on R. SGD needs to scale up a SGD, Thus,
a SGD
-((1))))))))))) or a SGD - a. SAM = (11(p1)))
(59)
SAM
In general, since [R] O(n), we can characterize the gap between a SGD and a SAM by Ol O(Vd/n).
E IMPLEMENTATION DETAILS
E. EXPERIMENT SETUP
We conduct major experiments on CIFAR- 100 (Krizhevsky et al., 2009) and ImageNet- (Rus- sakovsky al., 2015) using ResNer-50 (He al., 2016). We adopt pre-computed memorization scores for these two datasets from Feldman & Zhang (2020) to generate F of different memoriza- tion levels with [F| 5% We have | 3000 for CIFAR-I00 and [ 60000 for ImageNet. We sample high-memorization forget set Frigh by choosing [F] samples of highest memorization scores from S, Few by choosing [F] samples of lowest memorization scores, and Fiid by choosing [Fl samples whose memorization scores are closest to 0.5. We also run experiments with randomly sampled Frand on
and CIFAR-10 App. G. We use ReizedReeszeeCrrpp and HoaddomHorizontalPe as train transforms.
Pretraining and retraining. We pretrain on S and retrain on R with the same settings. For CIFAR- 100, we train for Ti = 200 epochs, use batch size 256, learning rate 1/o- = 0.1 with cosine annealing,
SGD with momentum 0.9 and weight decay 5 10-4 For ImageNet, we train for Ti = 150 epochs, use batch size 512, learning rate To 0.25 with cosine annealing and 5 warm-up epochs, SGD
with momentum 0.9 and weight decay 2 x 10-5, For CIFAR-10, we train ResNet-18 for Ti = 50 epochs, use batch size 256, learning rate T7 0.1 with cosine annealing, SGD with momentum 0.9
and weight decay x 10 4 We summarize the settings, test performance of different pretrained models, well accuracies of retrain models in Tab.
30

---

Published as a conference paper at ICLR 2026
Table 6: Pretraining settings and test accuracies using different A (top), as well as performance of retrained models w.r.t different F (bottom) for CIFAR-100 and ImageNet- IK.
Dataset, Model
Ir+warmup
Batch B
Epoch
W. Decay
SGD
A.AM
ASAM 1.0
SAM 0.1
CIFARIO0, Res50
0.1+0
256
200
5e-4
77.23
76.0
78.05 76.53
77.85
ImageNet, Res50
0.2545
150
2e-5
75.04
74.94
76.18
Retrain
High Mem
Mid Mem
Low Mem
Dataset, Model
Retain
Forget
Test
Retain
Forget
Test
Retain
Forget
Test
CIFAR100, Res50
99.964
3.3
74,96
99.981
57.5
74.14
99.956
100.0
75,81
ImageNet, Res50
97.134
13.828
74.826
97.388
52.27
74.832
96.671
99.858
75.018
Table 7: Ablation on weight mask cutoff choice for Sharp MinMax on CIFAR-100 with ResNet50. We report ToWs across different F and the averages. We observe that all choices work well: 10% works as well as 30%, while larger W as 50% can further improve the performance.
A=SGD
A=ASAM 1.0
Cutoff
Fhigh
Fiid
Flow
AVG
Fhigh
Fuid
Flow
AVG
10%
82.675
92.495
87.636
87.602
83.916
90.27
81.362
85.183
30%
82.27
94,913
86,504
87.896
84.521
87.761
84.381
85.554
50%
82.798
98.177
87.806
89.594
83.567
95.516
90.096
89.726
Unlearning. We conduct all unlearning methods for T2 10 epochs with the same batch size and optimizer settings. For NegGrad and Sharp MinMax, we unlearn with constant learning rate 0.02. We use a 0.99 for CIFAR-100 and a 0.989 for ImageNet accounting for its slightly smaller [F|/|S] ratio. For model splitting, we empirically find that small ratio for forget model benefits ImageNet such as 5%, while CIFAR- 100 suits a larger ratio such as 30%. For both pretraiming and unlearning, we wrap SGD with vanilla SAM (Foret et al., 2020) with p 0.1, and Adaptive SAM (ASAM) (Kwon et al., 2021) with p [0.1, 1.0], while keep other hyper-parameters the same for fair comparison.
F.2 SHARP MINMAX IMPLEMENTATION
Inspired by SalUn (Fan et al., 2023), we split the model into retain, forget models WR. W and update using two separate optimizers: SAM on WR and sharpness maximization on W F. We split the model by ranking the parameters that are important to F based on the magnitude of the gradient of the parameters after one pass on F,, and choose the highest percentage where we have 5% for ImageNet and 30% for 100. The cutoff choice is based on the wver-parameterization scheme: since ResNet50 w/ CIFAR- 100 is much more over parammarmettei than w/ ImageNet, there is less overlap between retain and forget parameters and more freedom to increase size of W for more aggressive unlearning. We have also experimented with 10% and 50% and notice a slight better performance of using 50% cutoff in Tab. 7. Unlike SalUn, which essentially performs RL unlearning on the selected parameters, we update both models using opposite optimization. SalUn also requires a larger part of the model to fine-tune with noisy, label flipped F (50%). We have summarized our implementation for weight masking in Alg. 1, and Sharp MinMax in Alg. 2.
E.3 UNLEARNING SETUP FOR PREVIOUS WORK
We compare with state-of-the-arr unlearning methods with optimized hyper-paramett settings. To our best knowledge, several previous methods are evaluated on ImageNet for the first time. We apply SGD and ASAM 1.0 on each 7/ and compare the performance between SGD and SAM. For LI-Sparse (Jia et al., 2023), we use unlearn Ir= 0.02 and 1x -4 For SCRUB (Kurmanji et al., 2023), we use unlearn Ir= 0.004, msteps= 8, kd_T= 4, 3 = 0.01, and y = 0.99. For RL (Graves unleam Ir= 0.06 CIFAR- 100 and 0.02 ImageNet. For SalUn (Fan al., 2023), we use the unlearn Ir= 0.06, 50% weight to finetune on CIFAR- 100, and unlearn Ir= 0.04, 30% weight to finetune ImageNet.
et al., 2021),
31

---

Published as a conference paper at ICLR 2026
Algorithm WeightMask
Require: forget_loades model, criterion, percent
1: for (name, param) model parameters do gradients[name] <- peros_iikepparall 3: end for
2:
4: for all (image, target) in forget loader do
5: loss + initerion(modelinnage target)
6: optimizer.zcro. sitpzerrzzzrorrrdd(() loss. baskwaak
7:
accumulate parameter gradients into gradients
8: end for
9: for all name in gradients do
gradients[name] < gradients[name]]
11: end for
vals <- cat ( ((flatten(v) | gradients. (nadientssvalues()))
cutoff f- quantile(all vals, percent)
> p.c.g., 0.1 bottom 10%
14: return name + (grad < cutoff) (name, grad) E gradients]
Algorithm SharpMinMax
Require: x retain, y y-retain, x forget y forget, model, criterion, mask, alpha, optimizer retain, op- limizer_forge"
1: r_lossl + a criitetion(modee(xxeeaan) retaii)
2: llossl.backward
3:
D. SAM first step
4: r_loss2 + a criterrin(/mode(xxrettii))) yetaain)
5: (askwab))
6: for (name, p) in model parameters do
7: if p.grad then
8: p.grad + p.grad @ mask[name])
D mask out forget grads
9: end if
10: end for
11: aetamiizerrrtain.sccooodd wwwp:/stteeoeoozdddttaiee
b sharp min
12: f lossl + -(-aa criterion (model(x forget), y forget)
f losst.backwardd]
14: Braimizer_forget.fisstsso. first_ wwt.sseeeeooorrada-aaaa
D. SAM first step
floos22 (1-a) siiterion(modee(xxfogge y fooret
16: f f.los22.acckwar(()
for all (name, p) in model parameters do
18: if p.grad then
19:
p.grad +- p.grad @ mask]name]
b update forget params only
20: end if
end for
22:
sharp
E.4 EVALUATION DETAILS
Membership inference attack. We adopted a MIA based evaluation from Jia et al. (2023). We train a binary classifier using the retain set R and the was involved in the training stage, based on the softmaxed outputs from the unlearned model. Then, we feed the forget set F to the classifier evaluate this unlearned model. We expect forget samples to be classified as "non-traie "non-training" data, and evaluate the unlcarning effectiveness based on MIA correctness. A lower correctness (close to 0.5) indicates difficulty to distinguish and thus better unlearning. This evaluation examines an unlearned model from privacy perspective.
Diest to distinguish whether data sample
Entanglement computation. We compute both entanglement scores based on normalized embed- dings of retain and forget sets from the penultimate layer of the model. We compute pisr-wise entanglement between each retain and forget embedding, either globally or within class. For variance-based entanglement F directly follow Zhao al. (2024) for implementation, and
32

---

Published as a conference paper at ICLR 2026
then rescale the raw scores to [0, based on the value range across global and class-wise scores. For Wasserstein entanglement Ew,, we randomly sample an equal number of embeddings from retain and forget embeddings and build two uniform proxy-distributions. We then use existing op- timal transport library to compute the transport distance (cost), outputting entanglement scores as 1 distance. No clipping needed
observe all scores lie within (0, 1].
F DETAILED EMPIRICAL RESULTS
F.1 STATISTICAL SIGNIFICANCE
We demonstrate the statistical significance of our main empirical results by running each unlearning experiment three times with different seeds. In Fig. 5 and Fig. 4, report the 95% confidence intervals (μ + 20r) of all unlearning methods on ImageNet and CIFAR-100, which correspond to Tab. 1 and Tab. 3. Each single bar represents the mean marked on top of its error bar plotted by =2a. We observe that SAM consistently improves all unlearning methods with more noticeable results CIFAR-100. For "All methods" subplots, highlight the largest improvement by applying SAM to each U. On CIFAR-100, we observe general larger variance of SGD based unlearning. especially for SCRUB. Despite that A=SAM 0.1 seems to provide weaker pretrained model, Adaptive SAM settings can improve unlearning performance more steadily with lower variance, which demonstrate that SAM unlearning more robust. Tab. 8 also records the means and variances of the "All methods" subplots for ImageNet and CIFAR-100. These additional insights further strengthen our findings.
runs and has the mean ToW scores
Table Verifying statistical significance (p main experiments on ImageNet and CIFAR- Given various pretrained model with different A, observe that SAM consistently improve base unlearn methods with higher means across multiple seeds. Moreover, we observe generally more stable performance with SAM based on smaller variance average.
ImageNet Method
RL
SalUn
NG
MinMax
SGD
ASAM 1.0
SGD
ASAM 1.0
SGD
ASAM 1.0
SGD
ASAM 1.0
A=SGD
82.9.10.3
83.910.2
70.6.10.1
71.0:0.11
83.5.10.3
84.8.10.0
80.2.1.11
87,910.0
A=ASAM 0.1
82.5+0.1
83.8+0.1
70.7+0.11
71.1+0.1
83.4=0.3
84.7+0.1
79.7+0.2
87.5=0.1
A=ASAM 1.0
83.2+00.
83.8+0.2
71.1+00.
71.2+0.0
84,1+000
84.6+0.2
80.1+0022
88.0+0.1
A=SAM
82.940.2
83.740..3
71.240.0
71.420.1
83.622.11
84.420.1
79.9=0.1
87.8-0.1
CIFAR100
LI Sparse
Scrub
RL ASAM 1.0
SalUn
NG ASAM 1.0
Method
SGD
ASAM 1.0
SGD
ASAM 1.0
SGD
SGD
ASAM 1.0
SGD
83880099
A-ASAM
63.6-177
69.3:0.6
54.3+118
79.3=0.88
72.1+0.9
75.8+1.3
72.91:1
82.5=0.4
83.9-088
85.5=066
AEASAM
64.2=0.7
68.72117
58.4110.5
72.0:2.1
75.711.5
80.31:12
79.0103
83.3=002
80.2:0.5
83.9=0.2
A-SAMO.1
68.31.0.6
41.111.7
19.71
74.240.7
79.411.
83.6::0.6
71.311.
78.7=005
F.2 COMPLETE ACCURACIES
In Tab. 9, Tab. 10, and Tab. 11, we report complete results of retain, forget, and test accuracies for all unlearning experiments, which used to compute ToW scores in Tab. I and Tab. 3. As have mentioned in the main paper, we observe that SGD often achieves lower test accuracies, motivating rethink the overfitting under sample-specific unlearning scheme.
G ADDITIONAL EXPERIMENTS
We provide additional experiments on CIFAR-10 10 and Tiny-ImageNet using randomly sampled for- get Frand To diversify our experiment settings, we use ResNet-34 with TmageNet-pretraiiee weights for our learning and unlearning
Tiny-ImageNet. Similar our main setup, we pretrain and retrain using the same settings, and we have summarized basic settings and baseline performance in Tab. 12. Since Tiny- ImageNet has 100K samples, set [Fandd 6000 for Imageen Tab. 13 records detailed accuracies and ToW scores various unlcarning and pretraining settings.
G.1 CIFAR-10
We summarize detailed unlearning settings on CIFAR-10. For LI-Sparse, we use unlearn Ir= 0.02 and 1 x 10.1 For SCRUB, unleam Ir= 0.004, msteps= 8, kd T= 3.5, B 0.01,
33

---

Published as a conference paper at ICLR 2026
Table 9: Detailed accuracies of NegGrad on ImageNet and CIFAR-100.
ImageNet
-SGD
AASAM 0.1
4-ASAM
AAASAM 0.1
Hlgh Mem
Retsin
Ferpet
Text
ToW
Retain
Forget
Test
ToWy
Retnin
Forget
Test
ToW
Recain
Forget
Text
ToW
+SGBD
88.766
25. 148
71.756
78.764
58.131
24.1
70,878
76.426
89.649
26.28
71.772
78.522
52.158
26.485
71.91
78.03
+,SAM0.1 0.1
89.487
26. -0.4
72.08
78.52
$8.640
24.77
20.983
78.36 38.366
89.767
26.542
72.276
78: 062
89.816
27:423
72.328
78.083
+ASAM 1.0
90804
28. 195
73.506
78.966
50. 00.3599
27.522
22.94
78.975
91.232
29 06.662
73.58
78, 09.0027
91.121
30.206
73.77
77 77.762
+SAM 0.1
910007
29.85
73.676
77.898
90,498
28,445
73.05
76.301
91,583
30.997
73.746
77.358
91.328
31,575
73,964
76.807
Mid Mem
Retain
Forget
Teat
ToW
Recoin
Forget
Test
ToWW
Rabsin
Forget
Test
TGW
Recain
Forget
Text
ToW
+SGD
88.771
56.87
71,414
$4.199
59.265
57,833
71.562
89.80
58.022
71,812
83.929
50.312
58.27
72.348
+ASA 0.1
89.56
58. 58.502
72.154
84.113
89, 69,276
57.698
71,576
84.07
90.087
59.08
72.378
84.267
89. 39.945
59,263
72. 22.482
84.062
+SSAM 1.0
90.999
61 998
73.544
83.389
91. 16.064
62.023
73.454
83. 83.358
914277
62:757
73.82
83.326
91.505
63.078
74.045
83 23.284
+SAM 0.1
91.395
63.015
73.734
$2.985
91.015
62.508
73.422
K3.04
91.984
64.367
74.014
$2.473
91.823
64.258
74.195
42.587
Law Mem
Retsin
Ferpet
Text
ToW
Remain
Forget
Test
ToW
Retsin
Furget
Test
TwW
Recain
Forget
Text
ToW
3GG
87.775
99.617
71,942
18.515
56,902
90.505
71,042
86
888847
663
72.41
80.
67.847
99.625
72-225
86.839
+SAM
65.251
72.19
68.256
99.635
72,064
89
72,658
89
50.
99
92.
90.042
+SAM 0.1
90.244
99
74.21
92,841
59.553
99,817
73.728
91.722
90.815
99.527
74,228
93.429
90.184
74.254
92.829
CIFARI00
A -SGB
--SAM 0.1
A -SSAM 1.0
A -SAA 0.1
High Mem
Retain
Forget
Test
ToW
Retsin
Rorget
Test
ToW
Retsin
Forger
ToW
Retain
Forger
Test
TVW
4SCDD
00.929
78.334
94.05
11.433
66.68
79 79.277
94.533
15.267
67.78
77:274
91.814
22.4
+ASAM
90.736
67.21
73.131
91,852
11.633
80.336
94.633
15.333
77.331
93.674
70.054
4ASAM 1.0
96.748
15,433
80.006
96-907
13.167
09.03
82. 82.196
96,99
17.3
69.85
78.731
96.376
24.033
69.85
72.518
+.
98,552
72.82
81.331
99.193
17.4
72.17
82.86
99.4
26.467
72.74
74.704
99.24
36.767
73.49
65.08
Mid Mem
Recain
Fergel
Test
ToW
Retsin
Forget
Test
ToW
Retain
Forget
Test
ToW
Retuin
Furget
Test
ToW
+SGD
90.162
02.
55.433
05.519
93.714
72.233
74.165
+ASAM
91,055
62.633
62.80h
95.005
58.133
95.524
93,838
72,367
+ASAM 1.0
96,781
69.533
6911
K1.465
97.16
64.391
97.199
72.7
69.58
79
97.257
76.2
69.8
75.653
+SAM0.1
98.938
50:133
72.18
75.059
90.007
76.133
70.87
77.04
99.448
85.1
72.59
70.808
99.160
90.033
72.9
66.089
Low Mem
Recain
Forget
Test
ToW
Ketain
Porpet
Test
ToW
Ketein
Forget
Test
TeW
Retain
Farget
'Jest
ToW
+SGD
91.085
97.767
83,718
952
66.637
93:117
85.307
96.933
76,374
+ASAM 8.]
92.736
90.767
67.3
86.78
94.670 676
58.55
87 97.671
94.290 25W
97.967
67.27
88.0339
86.902
96.9
62,92
78.087
+ASAM 1.0
92.834
07.8
67.53
87.052
96.267 267
99
68.0
90 502
97.888
99. 513
70.59
93.349
93.517
98.7
67.35
56. 759
+SAM
97.89
99.333
71.31
94.151
95.712
99.7
70.89
94.179
99.26
99 49.667
72.06
95.898
98.095
99.633
71.75
95.078
Table 10: Detailed accuracies of Sharp MinMax on ImageNet and CIFAR-100.
ImugeNet
A -SSC
A A-ASAM 0.1
AASAM
A ASAMM
High Mess
Retiin
Forget
Test
ToW
Retain
Forger
Text
12WW
Retain
Forget
Test
ToW
Retain
Forget
Tea
ToW
+SGU
$7.513
29.79
25.42
31.425
71.964
+ASAM
50,84155
67.991
72:234
58
65.712
Mid Mem
Retain
Forget
Test
ToW
Retain
Forget
Test
16W
Retsin
Forget
Fest
ToW
Retain
Forget
Test
TelW
-SGD
$7.089
91.5
71.418
50.881
86-757
58:373
71.1
80.754
87.217
50.0015
71.734
81 81.105
87.461
$9.677
71 31.848
80.913
-SAM 0.1
$6.936
50.585
71.35
$7.914
36.281
493533
70.514
87.40
87.561
$1.3
71.538
$5.039
87.529
52.043
71.84
87.642
-ASAM 1.0
$8.679
54.642
72.834
57.345
38.585
54.565
72.665
87.192
89.12
55.377
73.01%
87.27
89.002
55.733
73.192
87,076
+SAM
39.141
56.215
73.264
56.755
38.642
55.303
72.74
86.633
89.492
56.813
73.47
86.722
89.758
$7.657
73.792
86.486
Low Men
Ratuin
Forget
Trat
ToW
Retain
Freget
Text
T:WW
Ratain
Furgel
ToW
Retain
Forget
Tea
TUW
-SGD
55.798
99.51
71.644
56.334
84.345
99.482
70.594
84.37%
85.863
99. $9.568
71.61
86,002
85,098
99.57
71.45
55.517
-ASAM 0.1
$6,399
99.565 06.565
72.07
87.338
36.136
99.562
71.514
86.953
36.644
99,
72,104
57.554
85.894
99.593
71.395
86.668
-ASAM
$7,766
766
73.292
0944
87.365
99,77 077772
73.216
89, 138
96.159
99.
73,412
90.142
87.837
99.765 765
73.918
90,054
-SAM01
$7.856
777
73.666
90.005
87.745
99.00
73.55
89.852
86 06.
99: 765
73,94
91.111
87.974
99.39 792
73.752
90,207
CIFARI00
LSGD
-ASAMOL
-ASAM
ASAAM..
High Mem
Recain
Forget
ToW
Retain
Forget
ToW
Retain
Test
ToW
Retain
Furget
'esst
Too
+SGD
92.298
20.8
67.86
70.767
22.167
65.42
72.137
92.566
25.4
66.35
65.923
87.195
25.233
63.77
60.478
+ASAM UL]
39,574
6.133
66.57
7S,895
93.819
5.333
67.37
84. 966
92.095
6.3
66.52
81.825
86.909
9,253
64,33
72,897
+ASAM 1.0
92.121
6.467
67.15
82.27
976
5067
63.68
27.576
93,805
6.567
67.08
84.531
90:448
10.7
65.71
76.037
+SAM 0.1
97.383
7.1
11.61
90.378
98.185
6.133
71.04
91. 69.655
97.619
8.467
70.7
88.651
98,196
14,167
72.26
$5.195
Mid Mem
Rotain
Forpst
Retain
Forpet
ToW
Retain
Forpet
Retain
Forpor
ToWW
SGGD
91.433
66
60.79
76.692
91.633
63.367
64.39
77 264
92.11
69.4
65.56
74,526
85.714
62.6
62.55
71,951
+ASAM 0.1
91.16
42.7
65.88
96.027
91.4
40.333
64.11
96.451
95. 76
$1.3
66.74
93.33 03.786
88.074
55 35.867
63.61
50.104
+ASAM 1.0
92,585
16.9
66.81
94.913
94.074
43,133
33
99. 19.422
89.36
47 433
63.35
87.761
93.119
60.057
66.3
$3.633
97.453
97.874
95.
98.6
88
98.834
76.433
79.286
Low Mem
Retain
Porget
Retain
Forget
ToW
Rotain
Forget
ToW
Retain
Forger
Test
ToW
4SGD
50,579
97.6
66.09
67,853
80 781
97.1
64.36
81 847
88.605
07.833
64.78
80. 60.122
81.458
94.467
61.63
73.88 73.843
+ASAM 0.1
$9.026
95. 35.057
65.12
83.473
89.874
95.033
61.47
82.883
93.748
97.167
66.17
87.151
92,967
97.433
66.65
86, 66.659
4ASAM 1.0
91.951
96.567
56.504
92.819
91.131
84.581
85.014
62.79
77.461
+L
96.129
58.033
92.404
829
97.624
96.650
90.
and y 0.99. For RL and SalUn, we use unlear Ir= 0.08, and use 50% model parameters for SalUn. For NegGrad and Sharp MinMax, we use unlearn Ir= 0.02 and a = 0.99, and use 30% model parameters for unlearning on F and the rest for learning on R.
From the results in Tab. 12, we observe consistent improvement by using SAM except only two cases for RL and SalUn with A SGD. Surprisingly, Sharp MinMax is not the best algorithm CIFAR-10. 10. By the nature of its design to overfit to forget signals deliberately, we hypothesize that this approach might be aggressive for small-scale unlearning. We again observe SCRUB to be an
34

---

Published as a conference paper at ICLR 2026
Table II: Detailed accuracies of previous methods on ImageNet and CIFAR-100.
ImageNet
-SGD
ASAMM 0.1
-AAAMM
AA-AAM
Hlgh Mem
Retsin
Forpet
Text
ToW
Retain
Forget
Test
Toly
Retain
Forget
Test
ToW
Recain
Forget
Text
ToW
RL.
88.536
29:857
72.02
74,518
58.663
20,622
71,95
74.857
88.975
30.59
72.04
74.587
57.420
51.74
72.572
74.055
+ASAM 1.0
90884
33.395
74.234
74.951
90,615
32,868
73.972
75.221
91.14
34,745
74,298
73.95
91.155
35.333
74.322
73,579
SalUn
95.248
61.
75.04
44.981
93.016
65.807
74.976
46. 104
98.124
66.572
75.418
45.814
92.911
56.333
75.982
46-006
+ASAM 1.0
93113
66.217
75,496
45,998
92,965
65.058
75.25
06.938
93.134
66.472
75.712
45.856
92,855
06.033
76.172
46.358
Mid Mem
Retsin
Forget
Teat
ToW
Retsin
Fogge
T:at
1:W
Retsin
Forget
Teat
T6W
Rdiai
Forget
Teat
TaWl
RL
88,785
54.653
71.91
36.617
89,067
53.357
71.258
89.754
56.17
72,634
56.813
53.669
72.168
86.715
+ASAM
90.97
90.53
738836
85.581
50.457
50.337
73.654
85. 85.473
90,993
60.35
74.078
85. 509
90.907
60.402
74.348
85.494
SalUn
93.174
74,816
71,839
93,072
77.222
74.738
71.735
930078
77.118
75.362
72.308
92,825
77.167
75:365
72.419
+SSAM
93.098
77 08.983
75.47
71,554
92. 02.969
77.947
75.154
93.143
78.058
75.734
71 11.695
92.797
77.805
76,222
72.034
Low
Retain
Focget
Teat
ToW
Retain
Forpet
Test
1:W
Retzin
Furget
Test
Retain
Forget
ToW
RL
65,245
9% 603
71,162
$6.714
65,451
98.463
70.763
66.192
86.472
96.74
71.522
87.63
86.865
98, 95
72.36
88,594
+ASAM
99.408
75.728
91.069
88.214
99,377
90.42
88.965
99.457
58.963
99.500
91.74
SalUn
91.99]
99.77 778
76.612
55.008
91.743
59.77
74.463
94 652
91.696
99 31.
75.074
95. 116
91.412
99. 85
75.514
95.218
+ASAM 1.
92.095
99.85
75.224
95
65.628
91.882
99.818
74.992
95 224
91.967
85.
75.676
95
95.924
91,579
99.673
75 33.964
95.791
CIFARIO0
4S0D
AAAAMM 1.0
14-AM 0.1
High Mem
Resain
Forgel
Retain
Forget
Retaiin
Forget
ToW
Retain
Forget
TwW
LI-Spare
74.76
5.267
61,49
63.448
75.426
5.067
60.89
63 699
73,969
6.167
60.17
61 61.250
77,429
7,133
62.56
65 65,258
+ASAM 1.0
77,86
5.733
62.99
66.903
77.648
5.7
62.29
66.213
77.126
6.367
62.00
65.117
75,583
6.2
60.83
63.051
SCRUB
99.867
44.567
51.418
99.7955
73.85
67.163
99.902
45
74.59
57.816
99,971
76.47
43.246
+ASAM 1.0
99.962
50.313
95.055
99.971
18.137
52,681
68.
79.229
64.518
62.99
10:913
66
81,060
64.391
+ASAM 1.0
$4.012
9.7
63.85
69.952
81.519
61.41
66.909
86.195
63.53
69.73
89.324
65.99
72.884
Sa
39.024
16.567
09.926
86.298
15.467
62.71
66.541
91.207
20.7
67.355
90.598
18.533
09.671
+ASAM 1.0
94.557
20.9
68.96
73.268
92.126
18.3
65.94
71 4164
94.519
25.033
66.46
67.715
95.636
24. 367
68.80
70.933
Mid Mom
Retii
Forgel
Teat
TwW
Retoin
Forget
Test
ToW
Retsin
Forget
Test
TwW
Retain
Fonget
Text
TwW
Li-Sparse
67.864
68.686
71.305
38.633
72.775
68:264
37.933
68. 69.197
71,495
39.
59.73
71.94]
+ASAM 1.0
74.148
41.5
61.96
73.554
73.836
42.7
62.7
77.119
74.267
43.967
61.59
73.754
73,857
40.667
60.52
74.556
SCRUB
993864
91.4
74.29
76.125
99387
72.37
79.09
99.91
73.59
73.176
99.974
90. 90.167
25.76
08.433
+ASAM 1.0
99.974
85.133
75.51
73,353
99.969
77. 367
74.24
80 20.200
99.981
83.433
75.56
73.09
99.974
97 06.6
77.13
61.618
RL
79,262
37.067
62.53
84.395
75.757
31.733
58.31
80:215
81.955
36.433
61.31
86.411
81,906
38
61,48
85.481
+ASAM 1.0
$1,688
38.7
63.54
86.779
$1.086
37.355
62.3
86.357
85.674
38.7
63.05
91.124
86,914
40.167
63.08
38.033
Un
82,383
40.753
600.6
830056
82.4
40.9
00.9
N3.377
89.541
45 33.3
63.46
89. 39.700
90.205
46.857
64.8
50.895
+ASAM 1.0
91,579
48. 167
66.23
92.225
93.71
45.833
64.15
89.
187
94.117
50.5
66.77
93:401
94.2
52,333
67.91
92.914
Low Mom
Resain
Forgel
Test
TWW
Retain
Forget
Text
ToW
Retiin
Forget
Test
ToW
Retain
Forget
Text
TeW
LI Sparse
62.41
91.387
55.39
53.991
53.667
60.25
60.34
68.421
94.2
60.67
61.47
67.229
91,957
59.33
59,014
+ASAM 1.0
66.95
94.5
59.24
58.967
73.457
96.4
63.4
66 06.697
70.207
96.1
61.46
02.517
72.355
96.2
62.46
65.117
SCRUB
17.81
32.6
18.35
12.70%
15 606
28.367
15:87
10:33
66.324
90.167
56.04
5W.433
23.03
43.7
23.95
17.368
+ASAM 1.0
99.683
99.9
73.61
97.631
90.869
99.833
73.24
97. 508
99.64
99.8
73.7
97.776
99.729
99.8
73.77
97.933
RL
76,376
59.233
61.34
72.4
73.288
56.5
59.57
69.711
73.495
64.2
57.63
69.677
76.79
91.733
60.62
70.55
+ASAM 1.0
78,286
90.533
62,59
74.409
73,881
$7.3
59.08
09.375
82.095
89.333
03.53
80.321
83.483
94,167
64.12
78,166
SalUn
78.867
92.667
60.5
71.73
77.748
83.833
59.01
71.95
83.921
91.2
62.39
79.095
82.221
93.133
61,44
75.281
+ASAM 1.0
91,205
95.5
68.28
88.175
90.043
93.367
93.812
67.11
89.289
91,848
95,933
66.24
36.477
Table 12: Differed settings of pretrained models and their test accuracies using different A, as well as performance of retrained models w.r.t F Frand for CIFAR- 10 and ImageNe...
Dataset, Model
Irswarmup
Batch B
Epoch 7
W. Decay
SGD
ASAM 0.1
ASAM 1.0
0.
Retain
Forget
Test
CIFARIO, Res18
0.1+0
256
50
Se-4
93.02
93.26
93.38
99.943
92.567 59.383
92.49
TinylngNt, Res34
0.003+0
256
200
1e.3
62.1
62.77
62.74
63.87
99.985
61.69
unstable algorithm which collapses when unlearning with SGD given A = SAM0.1, while SAM helps reduce variance and stabilizes SCRUB unlearning given various pretrained models.
G.2 TINY-IMAGENET
We summarize detailed unlearning settings on Imay- For I.I-Sparse, we use unlearn Ir= 0.002 and a 1 1 For SCRUB, use unlearn Ir= 0.002, msteps= 8, kd_T= 3.5, 3 = 0.01, and 7 = 0.99. For RL and SalUn, we use unlearn Ir= 0.015, and use 30% model parameters for SalUn. For NegGrad and Sharp MinMax, we use unleam Ir= 0.005 and a 0.99, and use 10% model parameters for unlearning on F and the rest for learning on R.
From the results in Tab. 12, we observe consistent improvement by using SAM except few cases. SCRUB performs more steadily than on CIFAR-10. While RL and SalUn perform well on other datasets, they do not appear to be effective on Image
35

---

Published as a conference paper at ICLR 2026
Table 13: Detailed accuracies of previous methods on Tiny- Ima-eem and CIFAR-10.
Tiny Netgeeee
-SGD
ASAAMM
AAASAM
A-SAM
Random Proed
Retain
Forget
Test
ToW
Retain
Forget
Test
Retain
Forget
Test
T6W
Forget
Test
ToW
LI Sporse
79.247
52.235
49.61
74,009
62.722
56.217
50.81
77.545
$4.63
59,583
53.01
77.143
76.005
63.017
49.56
64,372
+ASAM L.0
89,379
59.5
54.37
82.753
90.81
60.933
54.35
$2,853
92.005
63.537
53.7
81.168
94.674
74.333
55:25
75,347
SCRUB
92.112
58.117
53.65
85.753
94.315
60.75
54.58
86,425
268
66.5
55.01
53.457
99.301
55.233
58.99
69.101
+ASAM 1.0
97,965
57.217
56.94
94.281
98.943
61.533
58.13
93,095
99.521
68.533
57.60
47.
99.962
97.267
61.05
61.704
RL
64,504
63.235
46.99
52.668
67.506
66.433
47.49
53. 144
70.319
69, 8883
48.
54,434
75,016
73.5
49.21
56. 56.397
+ASAM 1.0
69.356
68.133
49. 20
55,043
73,517
72,033
50,97
$7.345
75.88
75.617
50.36
36.384
81.006
79.683
50.94
$7.632
SalUa
69.39
66.45
50
55.735
70.057
68.767
49.54
55. 506
73.207
71. 78.
50.12
56,721
82.877
81.467
53.36
59.206
+ASAM 1.0
75,013
74.333
52.65
58,042
77.101
75.917
53,16
58.876
81.0.9
79.233
52.89
59.248
88.021
87,417
54.81
58. $8.996
NegGrad
$4,256
67.867
50.51
83,499
57.051
48.467
51.45
86, 060
86.575
52.0
51.26
168
99.979
99, 167
62.51
50,706
+ASAM 1.0
$0,907
50.45
54,47
91,594
93, 68)
51.35
53.66
93.094
96.343
54, 167
54.31
93.902
98.031
62.767
55.21
88.59
MinMax
$1.8
52.835
51.14
77.977
52.115
56,017
50,91
77.209
61.418
55,433
50.32
75.022
15.025
68.67
54.217
46.99
61.615
+ASAM
87,654
43.185
534
93.426
88.273
43.063
52.80
91.947
53.37
97.617
4H.5
96.466
CIFARIO
A-90D
A AAASAM
4-ASAM
1--AMM
0.1
Random Fooe
Retain
Forget
Test
ToW
Retain
Forget
Test
Retain
166
Porget
Tees
ToW
LI-Spunc
86.467
X2.967
82.25
85. 12
89,06
K5.567
54.45
87.688
86.683
83.466
K2.11
54.811
89.462
57.133
34.82
87.144
+ASAM LO
91,438
58.333
87.23
90.352
91,674
87.767
57.24
91,087
90.938
M5.7
M6.94
59.366
90.386
58.033
M6.43
N8.742
SCRUB
90.767
96.033
86.27
90.739
60,200
67.367
66,75
63.466
50. 1913
78.933
77.97
77.95
15.11
14.2
1.5
5.089
+ASAM 1.0
99.6
95.167
92.65
97.2
99.62 92
96.5
93,15
96.39
99 807
96.2
93.36
95.077
99.631
98.467
36.16
94.435
RL
92.774
56.6
87.22
93.116
90.569
14.2
65.17
91.02
91.445
34.133
K5.81
92.999
88.736
82.533
84.12
59.524
+ASAM 1.0
93,255
87,733
87.6
93.138
91,262
87.233
88,31
94,187
95 098
89,033
89
95.512
92:588
$6.561 567
87.4
93.206
SalUn
96.94
58.8
89.95
98.095
95. 726
87.6
59.02
97,052
95.5 99
58.733
89:35
596
96.612
89, $9.867
89.36 09.86
96 65.668
+ASAM 1.0
97,771
91.8
90.55
666
98. 26.24
91 168
91,41
97.917
98. 029
91.6
91.2
97.757
98.055
92,833
91:37
96.755
NegGrad
97,724
93.933
90,46
96.187
98.35
94. 06.
91.33
94,931
96. 024
94,267
90.92
94.9
96.405
93.4
89:72
93.009
+ASAM
95.83
99.219
96.2
92.
95,600
95.964
MinMax +ASAM
96.85
94.133
90.29
93.291
94.935
97.881
95.1
90.5
93.555
93.451
94.133
91.82
96.638
96.186
97.981
95.97
G.3 UNLEARNING WITH STRUCTURED NOISE
We consider noisy unlearning case where only corrupted version of S is available, following corruptions in ImageNer-C (Hendrycks & Dietterich, 2019) to apply glass blur and snow effect to CIFAR-100 100 with medium severity for additional empirical verification, and report ToWs in Tab. 14: We observe that SAM continues to improve base unlcarning methods with even more clear margins.
Table 14: Unlearning with ImageNet-C corruptions on CIFAR-100.
Glass Blur
A=SGD
A=ASAM
Method
High
Mid
Low
AVG
High
Mid
Low
AVG
NG
67.760
78.824
75.931
74.172
76.152
85,534
82,556
81.414
+ASAM
73.565
80.253
84.086
79.301
74.993
86.567
86,296
82.619
SharpMinMax
66.110
76.852
73.387
72.116
66.837
79.023
78.435
74.765
+ASAM
75.327
89.859
79.104
81.430
74.089
92.737
84.921
83.916
Snow
.=-GGD
A=ASAM
Method
High
Mid
Low
AVG
High
Mid
Low
AVG
NG
77.394
83.328
83.196
81.306
75.041
86.424
86,838
82.768
+ASAM
76.759
84.168
86.053
82.327
76.520
83.774
89.343
83.212
SharpMinMax
70.880
78.806
77.652
75.779
69.650
79.139
81.344
76.711
+ASAM
77.188
90.997
83,933
84.039
80.533
93.383
87.779
87.232
This is because that structured noise applying to the images affects the dataset's signal and noise vectors (4p and É.), causing a corrupted dataset with worse initial signal-noise ratio, but it does not affect update dynamics and the gained results under our theoretical framework, as corrupted images are still visually recognizable, and SGD still overfits more to the added noise.
G.4 SAM WITH ADAM AND VIT
We also verify that our observations generalize to different base optimizers and architec- tures. We experiment CIFAR-100 unlearning using ViT-Small (Dosovitskiy et al., 2020) and AdamW (Loshchilov & Hutter, 2017), and summarize our priliminary results in Tab. 15. For pre- training, use W with starting Ir 0.0001, weight decay 0.05, and set patch size to 4 for ViTlSmall on CIFAR-100 Other experiment settings are unchanged. For unlearning, we have un- learn Ir 0.0006 for NegGrad and for Sharp MinMax. Adam demands much smaller Ir than SGD and
36

---

Published as a conference paper at ICLR 2026
Table 15: Unlearning with ViT-Small and AdamW on CIFAR-100.
A=SGD
A=ASAM
High
Mid
Low
AVG
High
Mid
Low
AVG
NG
80.445
82.854
84.385
82.561
78.750
82.223
86.767
82.580
+ASAM
82.880
83.084
83.402
83.122
82.839
81.354
87.507
83,900
SharpMinMax
14.794
42.055
95.222
50,690
14.279
42.017
94.833
50.376
+ASAM
76.343
95.573
103.372
91.763
76.664
93.966
105.868
92.166
is more sensitive to unlearn Ir tuning. ViTs perform worse than ResNets on smaller datasets (test accuracies of pretrained models are 57%).
G.5 RELEARNING ATTACKS
We present relearning attack experiments in Tab. 16 to demonstrate SAM's unlearning robustness below. We take the unlearned models relearn the whole F for one epoch with small relearn- ing Ir, and measure the increase in forget accuracies. Reported are the averaged increase across nigh Fuid. Flow- We observe that SAM enhanced U are more resilient to relearning attacks with smaller increases. We note that these experiments highlight the robustness of our approach and hope that this encourages future works for deeper investigation into the role of loss landscape geometry for robust unlearning.
Table 16: Average increase of forget accuracies after relearning epoch F on CIFAR-100 and ResNet50. We observe that SAM enhanced unlearning is consistently more resilient to relearning attacks with less increase on forget accuracy.
Relearn 1==0002
Relearn 1==0.003
Releam 1r=0.004
A=SGD
A=ASAM
A=SGD
A=ASAM
A=SGD
A=ASAM
NG
8.644
10.333
11.167
13.256
12.789
14.7
+ASAM
8,533
9.289
11.033
11,533
13.022
13,389
SharpMinMax
13.1
15.067
15.589
17.5
16.144
18.511
+ASAM
8.333
8.8
10.667
11.2
12.711
12.667
7.122
8.556
8.5
9.589
9.622
10.989
+ASAM
6,222
7.378
7.444
8,489
8.367
9.467
G.6 RUNTIME AND EFFICIENT SAMS
We implement momentum SAM (MSAM) for unlearning CIFAR-100. As shown in Tab. 17, MSAM not only outperforms vanilla SAM by much less computation overhead but can also outper- form by average ToWs for some unlearning methods. This is plausible, recent efficient SAMs reduce computation with more informative perturbation directions than stochastic by momentum buffer, sparsity, prior gradients, sharpness-sensitive data, But there no clear theoretical jus- tification of MSAM rather than trying to stabilize the noise and reduce overhead of SAM. This warrants a deeper study beyond our scope our focus is to show superiority of loss landscape based methods for unlearning without worrying about speed (just like the original SAM paper), and we leave deeper theoreticallalgorihmii improvements and empirical evaluations for speedups for future work. We notice that while outperforming SGD) with much less computation overhead than vanilla SAM, MSAM does not outperform ASAM on SharpMinMax and SalUn. As we also observe that MSAM behaves differently from ASAM different forget sets, further and deeper investigation is needed to study the interactions between MSAM and weight masking to improve the performance. Our results have effectively demonstrated example of faster SAM variant that predictably ben- efits unlearning with less overhead.
G.7 KLOM SCORES
We follow (Georgiev et al., 2024) to compute KLoM of NegGrad with SGD and SAM and report the KL measures on F, R. Drea across F different difficulties, report means and -ereeeeenssssss in Tab. 18. Given pretrained model, we observe that SAM in unlearning also helps close the gap
37

---

Published as a conference paper at ICLR 2026
Table 17: ToWs of MSAM across different F in addition to reported results of baseline and SAM- enhanced unlearning on 100. We observe that MSAM not only costs much less computation overhead than SAM but can also outperform by ToW for some settings, since it leverages a smarter perturbation based on momentum buffer.
A=SGD
A=ASAM 1.0
Runtime
Frigh
Finid
Finm
AVG
Frigh
Fmid
Flow
AVG
L1-Sparse
63,448
68.686
53.991
62.042
61.252
68.197
61.47
63.64
165.3
+ASAM
66.903
75.554
58.967
67.141
65.117
73.754
62.517
67.129
323.6
+MSAM
68.768
76.378
64.932
70.026
70.885
76.342
65.068
70.765
201.3
NG
78.334
83,335
83.718
81.796
77.274
78.59
85.443
80,436
309.5
+ASAM
80.806
81.465
87.052
83.108
78,731
79.264
93.249
83,748
610.3
+MSAM
81.811
85.568
91.176
86.185
73.291
77.43
91.691
80.804
352.4
SharpMinMax
70.767
76.692
82.853
76.771
65.925
74.526
80.127
73.526
+ASAM
82.27
94.913
86.504
87.896
84.521
87.761
84,381
85,554
631
+MSAM
79.079
73.057
88. 18.7
80.098
77.034
72.944
93.819
81,266
398.1
RL
68.464
84.395
72.4
75.086
66.689
86.411
69.677
74.259
173.3
+ASAM
69.952
86.779
74,409
77.047
69.73
91.124
80.321
80.392
344.1
+MSAM
73.032
87.608
76.537
79.059
72.656
90.675
81.027
81.453
216
SalUn
69.926
83.056
71.73
74.904
67.355
89.768
79.095
78.739
172.8
+ASAM
73.268
92.225
88.175
84.556
67.715
93.401
89.289
83,468
340.9
+MSAM
70.011
89.214
81.069
80.098
68.548
92.289
82.757
81.198
213.6
between unlearned models and retrained models, when the standard way retrain models with SGD does not favor SAM as they adopt different optimization dynamics. We observe similar trends measuring performance closeness with ToWs: while SAM does not improve KL closeness F, it reduces the KL divergence on R and Diest and often halves KL on R. Adding SAM (vs. SGD) reduces the distance the retrained reference across memorization levels; e.g., with SGD-retrained reference the average distance drops from 0.0973 w/ SGD to 0,0827 w/ SAM. The reported KL at 95%- 55%-eeecentsles in the second table also show that SAM reduces KL even tails (smaller variances). We use N 10 with 10 bins for our LLoM measurements (pretraining 10 models and retraining 30 models for each F, and unlearning 120 models for all settings).
Table 18: Mean and 95%-percentile KLoM on CIFAR-100 and ResNet50 after NegGrad unlearn- ing. [SGD, SAM] denotes SGD-pretrained and SAMMunlearned. We observe that SAM enhanced unlearning consistently improves KLoM across different F. Similar to what we observe with ToWs, SAM performs better on retain and testset. Based on perceneeee KLoM scores, we observe that SAM enhanced unlearning consistently improves KLoM on tails too, which also indicates the better stability of SAM unlearning.
KLoM Mean
A=SGD
A=SGD Retain
A=SGD
AVG
A,
Forget
Retain
Test
Forget
Test
Forget
Retain
Test
SGD, SGD
0,1294
0.0721
0.1293
0.1221
0.0669
0.1259
0.0284
0.0747
0.1271
0.0973
SGD, SAM
0.1384
0.0411
0.1076
0.1989
0.0331
0.0983
0.0163
0.0411
0.1093
0.0827
SAM, SGD
0,1549
0.0676
0.126
0.1313
0.066
0.1246
0.025
0.0658
0.1248
0. 1007
SAM, SAM
0.1714
0.0264
0.095
0.2457
0.0253
0.0913
0.015
0.0311
0.1008
0.0891
KLoM 95%
A=SGD
A=SGD
A=SGD
AVG
A,U
Forget
Retain
Test
Forget
Retain
Test
Forget
Retain
Test
SGD, SGD
0.3397
0.2097
0.5212
0.4959
0.2097
0.4901
0.0956
0.2097
0.4959
0.3408
SGD, SAM
0.3397
0.2097
0.4901
0.5681
0.0956
0.3723
0.0956
0.2097
0.4901
0.319
SAM, SGD
0.4901
0.2097
0.5212
0.5371
0.2097
0.4959
0.0956
0.2097
0.5212
0.3656
SAM, SAM
0.4901
0,0956
0.3723
0.8111
0.0956
0.3397
0.0956
0.0956
0.4901
0.3206
38

---

Published as a conference paper at ICLR 2026
H COMPLETE VISUALIZATIONS
In this section, provide complete visualizations of feature space and loss landscapes of pretrained models, NegGrad unlearned models, and Sharp MinMax unlearned models, comparing SGD with SAM across all memorization levels. The observations are generally consistent across memorization levels, with Fligh being more noticeable.
H.1 LOSS LANDSCAPE
Inspired by Wu et al. (2017), we quantify the flatness by basin ratio, which is the percentage of perturbed losses whose deviation from original loss I 0.5 stddev. Fig. shows loss landscapes of Des and Fhigh- We observe SAM has higher basin
SAM and SGD before and after unlearning ratios (flatter landscape) than SGD for pretrained model and MinMax unlearned model as expected. Surprisingly, SGD can become flatter after unlearning. We conjecture that the gradient ascent might be implicitly regularizing SGD which had more overfitting than SAM during pretraining. We leave the further characterization of loss landscapes to future work.
H.2 FEATURE VISUALIZATION
39

---

Published as a conference paper at ICLR 2026
A-SGD
A=ASAM 0.1
+7.9
88.0
A=ASAM 1.0
+1.3
84
84.5
%0.9
63.93 13.33.4 33.6
ToW (96
80. 10
74.1
70.00, 71.112
FLO1VI 21,
70.6
RL
ASAMI 1.0
SalUn
ASAM 1.0
NG
ASANI 1.0
MinMax
ASAM 1.0
(a) All methads
A-ASA 0.1
A-ASAM 1.0
B4.7
84.6
84.4
$4.4
ToW (96)
84.0
84.0
83.9
83.6
83.5
63.4
83.4
5/5GG
UISASAM 0.1
USSASA 1.0
1/~ 0.1
(b) NegGrad
A=SGD
A=ASAM 0.1
A=ASAM 1.0
88.1
A-SAM 0.1
87.9 I
87.5
88.0
87.8
T
I
87.4
86.4
ToW (96)
84.7
83.1
81.4
86.2
80.1
79.7
79.7
1>56D
1/5ASAM 0.1
ASAM 1.0
UN-AMM 0.1
(c) Sharp MinMax
Figure 4: 95% confidence intervals (μ + 2x) of unlearning methods on ImageNet, in accordance to Tab. and Tab. We run each setting three times with different seeds and compute the statistical significance. SAM consistently improves base U, and we observe ASAM 1.0 to bring largest im- provement steadily.
40

---

Published as a conference paper at ICLR 2026
85.5
A=ASAM 0.1
A-SGD
+3.6
63.5
825
+7.3
92.0
A=ASAM 1.0
+25.0
79.3
+6.1
80.00.1
75.7
72.8
76
76.6
74.2
742
72.0
+5.7
TOW (96
9.9
58.9
50.0
ASAMI 1.0
Scrub
ASAM 2.0
RL
ASAM 1.0
SalUn
ASAAM 1.0
NG
ASAM 1.0
L1
(a) All methods
85.5
84.0
A-ASAM 0.1
84.00
83.99
A-ASAM 1.0
I
#1.0
79.8
ToW (96)
78.7
77.0
75,5
74.2
73.8
71.3
D/5GG
D/SASAM 0.1
ESSASAM 1.0
2/NSAM 0.1
(b) NegGrad
A-SGD
92.7
A=ASAM 0.1
92.0
A=ASAM 1.0
A-SAM 0.1
87.8
85.6
82.7
B4.6
ToW (%6
78.1
77.5
74.3
73.2
68.3
1>55D
1/SASAM 0.1
ASAM 1.0
UNSAM 0.1
(c) Sharp MinMax
Figure 5: 95% confidence intervals (μe + 2x) of unlearning methods on CIFAR-100, in accordance to Tab. and Tab. 3. We each setting three times with different seeds and compute the statistical significance. SAM not only improves ToW of the based methods, but also more robust against variance than SGD.
41

---

Published as a conference paper at ICLR 2026
Basin:
Basim: 40
Basin:
Basin: 44.32
Basim:
Basin: 28,
D Diesi A = SAM
Dist, NG
D MinMax
highh A SAM
F higgh NG
Frigh. MinMax
Basin:
Basin: 41.44
Basin:
Basin: 37.92
Basin: 46.88
Basin: 22.08
Dien: A = SGD
Dist. NG
Diess MinMax
Fligh A = SGD
Fiigh, NG
Frihh MinMax
(a) High Mem Fhigh-
Basin:
Basin: 46
Basin: 50.40
Basin: 29,
Dieni A = SAM
Dist, NG
Duse. MinMax
Fiid A SAM
Fmid. NG
Fuid. MinMax
Basin:
Dien, A SGD
Diss. NG
Diss. MinMax
Fmid, A SGD
Fmid. NG
Fmis, MinMax
(b) Mid Mem Fmid.
50.24
Basin:
Basin: 66.08
Basin:
Diest A SAM
Dust, NG
Duse. MinMax
Fun , SAM
Fio., NG
Fiow, MinMax
37.44
Basin: 40.16
Dien: A = SGD
Dus, NG
Duss. MinMax
Few. SGD
Fuw., NG
Fioe. MinMax
(c) Low Mem Fow.
Figure 6: Loss landscapes of SAM and SGD on Diest and all F. As memorization level goes down, F becomes easier to unlearn and SGD shows less to no "regularizing" effect as we have discussed on Frigh. The general trend preserves with decreasing memorization levels and SAM is generally flatter before and after unlearning.
42

---

Published as a conference paper at ICLR 2026
Tore
Target
SAM pretrained, E
55.86, Bw 45.45
SGD pretrained, E
59.58, Bw 51.21
Clases
Fasan
Resh
Fash
SAM NegGrad, EW
49.87, EC:
36.42
SGD NegGrad, EW,
51.24, E P
35.22
All Clacous
Davoes
Notin Fore
SAM MinMax, EAIl 51.26, E 33.65
SGD MinMax, EW, 51.12, ECh 38.41
Figure 7: UMAP (MeInnes al., 2018) feature analysis on High Mem Fuih We observe SGD unlearning forms more obvious clump in all-classes panels while SAM unlearning better maintains class clusters. From classwise panels, we observe that SAM effectively pushes forget samples away while gathering retain samples to dense cluster, while SGD also scatters retain samples during unlearning, suggesting overfitting.
43

---

Published as a conference paper at ICLR 2026
Beain
SAM pretrained, E
61.74 BW 49.88
SGD pretrained, EW
66.3, bW
57.11
Cliss
SAM NegGrad, EW,
52.36, E 14.71
SGD NegGrad, EA Davee
52.99, E 46.91
All Davoes
Class
SAM MinMax, EAIL 51.8, EW, 44.56
SGD MinMax, EW, 53.7, EW, 49.57
Figure 8: UMAP (MeInnes et al., 2018) feature analysis on Mid Mem F At all-class level, observe that SAM better maintains class clusters after unlearning while SGD is forming a more evident clump of features; at classwise level, we observe that while both push away forget features, SGD also scatters retain features further, suggesting overfitting. This also explains the larger clump of SGD all-class level.
44

---

Published as a conference paper at ICLR 2026
Deain
SAM pretrained, EW
59.84, BW
52.46
SGD pretrained, EW
63.13, B
59.64
Cliss
SAM NegGrad, EW 54.93, EW, 50.83
SGD NegGrad, EW,
56. 12, E P
55.93
All Elacuus
Dassues
wan Poge
SAM MinMax, EAIl 55.08, EN 52.55
SGD MinMax, EW, 56.77, ECh 57.15
Figure UMAP (MeInnes al., 2018) feature analysis on Low Mem Flow. We observe SGD unlearning forms more obvious clump in all-classes panels while SAM unlearning better maintains class clusters. From classwise panels, as Fio requires less unlearning and the model can generalize to Fkow forget samples do not move much expected. But on SGD MinMax we still observe that SGD scatters more retain samples away.
45


Reviews:
[{"review_id": "XJG5E7dlrr", "reviewer": "Reviewer_pWni", "summary": "The paper investigate the idea of sharpness aware optimization for the machine unlearning, they characterize SAM under NegGrad unlearning and the theoretical study of bounding and choosing weight factors. The paper provides a comprehsive theoretical proof to show that SAM can acheive successful unleasrning with significantly smaller damage to the retain accuracy.", "questions": "The memorization classes F(high) etc. do not seem to follow a linear pattern in the ToW scores. For instance sometimes mid has the highest score, sometimes mid has the lowest score. Why should we believe that memorization is a worthwhile way to classify these different forget sets if they don't have a clear correlation to increasing difficulty?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Theoretical analysis of SAM is robust. The paper has done a good job of analyzing it in relation to Random Labeling and especially NegGrad. I appreciate they were honest about SAM+NegGrad giving worse forget accuracy than SGD+NegGrad before their discussion of overfitting.", "weaknesses": "1. UMAP Visualization Clarity\nThe UMAP visualizations are difficult to interpret. For instance, Figure 1 is intended to illustrate inter- and intra-class movements after unlearning; however, these differences are not visually discernible. The colors used are too similar, and the expected variations are not immediately perceptible. I recommend improving visual clarity by adopting a more distinct color palette, varying marker shapes, or explicitly highlighting changes using arrows, circles, or other annotations. This would make the patterns of change more evident to the reader.\n\n2. ToW Score Chart Interpretation\nWhile the chart presenting the ToW scores supports the claim that SAM is an effective optimization method, it lacks fine-grained detail. Including additional breakdowns or complementary metrics could help the reader better understand how SAM contributes to performance improvements beyond the aggregated ToW score.\n\n3. Reporting Underlying Accuracy Metrics\nThe paper notes that SAM+NegGrad achieves higher forget-set accuracy than SGD+NegGrad, but these results are only reflected indirectly through the ToW score. It would strengthen the analysis to report the individual accuracy components that contribute to ToW alongside it. Relying solely on a composite metric can obscure nuances in model behavior; providing detailed accuracy values would enable a clearer assessment of the unlearning method’s performance.\n\n4. Lack of Statistical Evaluation\nThe reported results appear to lack statistical evaluation—no mean or standard deviation values are provided (e.g., in Tables).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JmEEIP02o", "reviewer": "Reviewer_U1T3", "summary": "This paper revisits Sharpness-Aware Minimization (SAM) in the context of machine unlearning (under the NegGrad unlearning framework) and demonstrates that SAM exhibits a desirable property in distinguishing between the retain and forget sets, as the forget set tends to overfit. The analysis is conducted using a simple two-layer CNN setup. The authors further observe that incorporating SAM into NegGrad allows effective unlearning with a smaller ratio between the retain and forget sets. Experimental results consistently show that SAM-augmented methods outperform their counterparts that do not employ SAM regularization.", "questions": "Where equation 8 is used in the proposal eventually? Is it used to train on forget set?\nWhy is the discovery of smaller ratio $\\alpha$ such useful? Is it generalizable to other unlearning algorithms? or those with gradient descent/ascent only?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper provide in-depth theoretical analysis on how SAM would impact the unlearning quality.\n2. Experiments contains sufficient evidence on how well SAM argumented model performs compared to the original classic unlearning methods.", "weaknesses": "1. The paper is not well polished and contains numerous grammatical errors and excessive notation, which makes it difficult to read. While analytical papers often introduce complex notation to convey ideas precisely, this paper’s presentation suffers from a combination of grammar issues, overly long sentences, and vague explanations. A few specific examples:\n    1. Equation 8 and corresponding description is very vague, and I cannot get how this equation is used in the proposed work. Is it on forget set right? How it connects to all the description in 3.1? There is no sufficient connection given here. \n    2. Line 115: Each image consists of ..... This sentence is long, contains grammar error, and it is not parsable. \n    3. Definition of noise coefficients $\\xi_{j,r,i}^{(t,b)}$. I got totally lost with such type of notions. I think it is unnecessary notation complexity introduced; without them, the paper reads fine.\n\n2. The use of Sharpness-Aware Minimization (SAM) in unlearning is not new. The claimed contribution of this work is therefore unclear. If the main purpose is to provide analysis, the key takeaway appears to be that SAM allows unlearning with a smaller retain set. However, in practice, the exact ratio of the retain set is not a major bottleneck in most unlearning scenarios. The authors should justify why this analysis provides meaningful new insight or practical benefit compared to prior work.\n\n3. All experimental results are reported solely using the ToW metric, which does not appear to be a standard evaluation measure for unlearning. While introducing a new metric can be valuable, the absence of established unlearning metrics (e.g., retention accuracy, forgetting efficacy, or relearning resistance) raises concern. It remains unclear whether SAM genuinely improves unlearning performance in practice. A broader evaluation would be needed to substantiate the claimed advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gZmBZOqipV", "reviewer": "Reviewer_F8sS", "summary": "This paper presents a theoretical and empirical investigation of machine unlearning through the perspective of sharpness-aware optimization. The paper studies how SAM, originally designed to improve generalization, behaves under unlearning scenarios that involve conflicting forget and retain signals.", "questions": "1. In Section 3.2, line 290-294: Could the authors explain the motivation of this weight mask design, and why it is different from SalUn [1]?\n2. I think a rather important baseline is missing in the experiments: performing SAM on the retain loss only. According to the paper, this should improve retain performance without degrading forget performance. Additionally, this can be more efficient in run-time compared to Sharp MinMax. By comparing this baseline to Sharp MinMax, the authors can show the effectiveness of sharpness maximization on the forget loss. Could the authors explain why they did not include this baseline in the experiments?\n\n3. Could the findings in this paper be extended to machine unlearning in image generation tasks [1]? This could greatly enhance the soundness of the proposed method. \n\n[1] https://arxiv.org/pdf/2310.12508", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors develop a signal–noise decomposition framework to analyze how SAM and SGD behave when learning (retain) and unlearning (forget) signals interact, giving novel insights on how SAM affects unlearning performance.\n2. The paper further proposes Sharpness MinMax, a novel optimization method that improves machine unlearning performance.\n3. The effectiveness of the proposed method is evaluated on multiple datasets. The paper also gives additional evaluations such as feature entanglement and privacy evaluations.", "weaknesses": "1. For the evaluation metrics, run-time evaluations are missing. Employing SAM on both forget and retain loss might significantly increase run time, and this efficiency trade-off should be evaluated.\n\n2. Evaluations of unlearning robustness are missing, such as evaluating the performance of the unlearned models against relearning attacks [1]. Sharpness-aware minimization employed in LLM unlearning [2] increases the robustness of the unlearned models against relearning by perturbing the model with a 'worst-case' perturbation at each optimization step. In light of [2], does Sharp MinMax decrease the robustness of unlearning by maximizing the sharpness of the forget loss? I think this concern should be studied in this paper. \n\n3. There are limited explanations on the weight mask design for Sharp MinMax. The authors use 5 % cutoff for ImageNet and 30 % cutoff for CIFAR‑100. However, it seems that there are no explanations or ablation studies on this.\n\n4. Please refer to the questions for additional concerns.\n\n[1] From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization\n\n[2] Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uu2HNiELSs", "reviewer": "Reviewer_Ti1T", "summary": "This paper proposes using SAM for unlearning. They provide theoretical evidence in support of their proposition as well as experimental evaluations that substantiate their claims. The authors also point out that unlearning difficult examples is facilitated by overfitting and as a result the propose Sharp MinMax, a novel algorithm that utilizes this observation.", "questions": "My main questions involve the weaknesses that I issued above.\n\n1. In [1] the authors introduce as a metric for successful unlearning KLoM. Does SAM achieve an improved KLoM when using Negative Gradient in comparison to the SGD alternative?\n2. Does SAM resolve the issues raised about data dependencies between retain and forget as raised in [2]?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper has several contributions.\n\nFrom a theoretical standpoint:\n\nThey show that for the negative gradient unlearning method one can minimize the test error for SAM always under proper assumptions, while this is not the case when someone uses SGD, which is interesting.\n\nFrom a heuristics standpoint:\n\nBased on their theoretical findings and that SAM fails to remove noise for the forget set unlike its behavior for the retain they suggest a sharp minimax algorithm with the purpose of overfitting to the minima in order to unlearn the samples of the forget set effectively. The idea of overfitting is in general an interesting proposition.\n\nFrom an empirical standpoint:\n\nThe paper provides extensive experimental evaluations of the methods over multiple datasets. With respect to the ToW metric they provide SOTA performance, making the empirical findings substantial in support of their claims.\n\nThe paper also provides a study on the effect of the weighting $\\alpha$ of the negative gradient method, for SGD and SAM based methods which can be beneficial for the community.", "weaknesses": "The main weakness that I find in the paper is an assumption that is made implicitly but yet not discussed substantially. The assumption is the following: \"Conversely, a sample i ∈ F of class j, which we want to predict −j in unlearning\" as stated in line 217. Although this is some times the case a lot of the time it is not and recent works [1],[2] have demonstrated that empirically and theoretically respectively this assumption will lead to an \"illusion\" of unlearning where the model does not behave as an oracle model that has never seen the data in the Forget set. This can be remedied with a relevant discussion or a clarification as there are unlearning settings where this assumption actually holds necessarily by definition as they clarify in [3] for the RB or RC unlearning scenarios.\n\nDespite the aforementioned weakness which is inherent in some part of the unlearning literature, in my opinion, I find the idea that one unlearns by overfitting in the retain or through SAMs property of dealing with noise in the retain but not in the forget intriguing.\n\nAdditionally I would like to mention a minor weakness in my opinion which is less significant and deducts from the presentation of the work. At line 470 the authors state that the experimental evaluations verify their claim. Since the claim about $\\alpha$ does not have an extensive proof I believe that the correct term would be that it supports, instead of \"This verifies our claim that α depends more than retain-forget ratio.\"\n\n[1] Attribute-to-Delete: Machine Unlearning via Datamodel Matching, Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, Seth Neel\n\n[2] Ascent Fails to Forget, Ioannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi, Volkan Cevher\n\n[3] Towards Unbounded Machine Unlearning, Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, Eleni Triantafillou", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)