Unlearning-Aware Minimization
Authors: Hoki Kim
Keonwoo Kim
Sungwon Chae
Sangwon Yoon
Venue: NeurIPS 2025 poster ()
Keywords: Machine Unlearning
Min-Max Optimization
Unlearning-Aware Minimization
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper introduces Unlearning-Aware Minimization (UAM), a novel min-max optimization framework for effective machine unlearning, which outperforms existing methods on image classification and text-based question answering datasets.

Abstract: Machine unlearning aims to remove the influence of specific training samples (i.e., forget data) from a trained model while preserving its performance on the remaining samples (i.e., retain data). Existing approximate unlearning approaches, such as fine-tuning or negative gradient, often suffer from either insufficient forgetting or significant degradation on retain data.
In this paper, we introduce Unlearning-Aware Minimization (UAM), a novel min–max optimization framework for machine unlearning. UAM perturbs model parameters to maximize the forget loss and then leverages the corresponding gradients to minimize the retain loss. We derive an efficient optimization method for this min-max problem, which enables effective removal of forget data and uncovers better optima that conventional methods fail to reach.
Extensive experiments demonstrate that UAM outperforms existing methods across diverse benchmarks, including image classification datasets (CIFAR-10, CIFAR-100, TinyImageNet) and multiple-choice question-answering benchmarks for large language models (WMDP-Bio, WMDP-Cyber).

Unlearning-Aware Minimization
Hoki Kim't Chung-Ang University hokikim@cau.aa ac.kr
Keonwoo Kim NAVER Digital Healthcare LAB
keonwoo wwwul/@naverrrrrcooo com
Sungwon Chae Seoul National University csw08150snu, kc.
Sangwon Yoon
Ministry of Justice, Republic of Korea asd010752727500gmai1] com
Abstract
Machine unlearning aims to remove the influence of specific training samples (i.e., forget data) from trained model while preserving its performance on the remaining samples (i.e., retain data). Existing approximate unlearning approaches, such fine-tuning or negative gradient, often suffer from either insufficient forgetting or significant degradation on retain data. In this paper. introduce Unlearning-Awe Aware Minimization (UAM), novel min-max optimization framework for machine unlearning. UAM perturbs model parameters maximize the forget loss and then leverages the corresponding gradients to minimize the retain loss. We derive an efficient optimization method for this min max problem, which enables effective removal of forget data and uncovers better optima that conventional methods fail to reach. Extensive experiments demonstrate that UAM outperforms existing methods across diverse benchmarks, including image classification datasets (CIFAR-10. CIFAR-10M 100, TinyylmgeNNet and multiple-choice guestion-answeriigg benchmarks for large language models (WMDP-Bio, WMDP-Cyber).
Introduction
The increasing deployment of artificial intelligence (AI) into real-world applications has prompted critical discussions regarding the alignment of Al with human values. A key aspect of these discussions General Data Protection Regulation
the "right to be forgotten" " (25). a principle embedded (GDPR) [14. This right enables individuals to request the deletion of their personal data, providing safeguard for privacy and mitigating risks associated with data misuse.
The simplest approach is retraining a model from scratch without the data points to be forgotten. However, retraining computationally prohibitive, particularly for large-scale deep learning models 4. 33). As a potential solution, machine unlearning has emerged for removing the influence of specific data samples by appropriately updating their parameters 2 3 9. unlearning is to efficiently remove the influence of specific data while preserving performance on the remaining data. Formally, let D, denote the dataset to be forgotten forget data) and D, denote the dataset to be retained (i.e., retain data). A common strategy of machine unlearning is to optimize the following objectives: minimizing the retain loss C(w, D,)) or maximizing the forget loss C(w, D 32
The goal of machine
In this paper, introduce Unlearning-Aware Minimization (UAM), a novel min-max optimization framework for machine unlearning. Specifically, UAM formulates unlearning as two-stage process:
"Corresponding author: hokikim@cau. kc.
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

3
1.0
1.0
Retain Loss
FT
0.8
0.8
NG
Forget Less rlw. DD.
Retain Loss E(w. .))
UAM
1
0.6
0.6
W2
0.4
0.4
NG
Forger
0.2
NG
0.2
0.0
UAM
0.0
-3
0
100
200
300
0
100
200
300
Iteration
Iteration
(a) Optimization trajectories
(b) Forget loss L(w,, Dy) and Retain loss L(w, D,)
Figure 1: Optimization results on synthetic loss functions. The optimization begins at the global
minimum of the sum of two losses (black cross, +). (a) UAM successfully converges the optimal point (yellow star, *), whereas other methods converge to suboptimal regions. For NG, only partial trajectory is shown due divergence.
(b) UAM achieves high forget loss while effectively minimizing the retain loss. NG yields a high forget loss but fails to maintain low retain loss. FT converges a suboptimal region, resulting in lower forget loss.
(i) an inner maximization that identifies a surrogate weight within a local neighborhood that maximizes the forget loss, and (ii) outer minimization that reduces the retain loss by using its gradients. Intuitively, this procedure ensures that model exhibits characteristics similar to those of weights with high forget loss, while maintaining low retain loss. By leveraging first-order Taylor approximation, we derive a scalable algorithm that enables effective unlearning while remaining computationally practical.
Fig. T highlights the key differences between UAM and existing unlearning methods. Fine-tuning (FT) minimizes L(w, D,) and negative gradient (NG) maximizes L(w,, Dy) [1]. Both methods converge to suboptimal regions the given optimization problem. These methods result in either low forget loss (i.e., insufficient forgetting) or high retain loss (i.e., poor performance). In contrast, UAM more effectively navigates the loss landscape, approaching the optimal solution characterized by high forget loss and low retain loss. As shown in Fig. [b6] UAM achieves higher forget loss and a lower retain loss by explicitly exploring regions with high forget loss. In our experiments, UAM demonstrates superior performance on both image classification and multiple-choice question answering tasks.
Our main contributions can be summarized as follows:
We propose a new min-max optimization framework for machine unlearning, Unlearning- Aware Minimization (UAM). By leveraging model parameters with high forget loss, UAM enables the effective removal of forget data while preserving the performance on retain data.
We establish an efficient algorithm based first-order Taylor expansion. We also provide a theoretical analysis of UAM, characterizing its optimization dynamics through the cosine similarity between retain and forget gradients.
We evaluate the effectiveness UAM on image classification datasets (CIFAR-10. CIFAR- 100, and TinylmageNet) and multiple-choice question-answering datasets (WMDP and WMDP- Cyber) using large language model (LLM). Since UAM a framework independent of any specific loss function, it can be easily extended to other domains.
To promote reproducibility and benchmarking within the machine unlearning community, we release implementations of existing baseline unlearning methods, along with proposed framework, available at: sittps://ae thub. com/ com/Harry24k) machine eeariing-pytorchh

---

2 Related Work
2.1 Machine Unlearning
aims to eliminate the influence of forget data Dy. while preserving knowledge learned from retain data D,. The ideal solution, known exact unlearning, is to retrain the model from scratch without Dy; however, retraining is often computationally inefficient for large- scale deep learning models 4 33]. Therefore, approximate unlearning methods have been developed. Fine-tuning (FT) 32] minimizes L(w, D,) relying catastrophic forgetting [24]: negative gradient
Machine unlearning [ [
(NG) [9, which also referred as gradient ascent, directly maximizes [(w, Dj). More recent methods build on these frameworks by incorporating additional techniques such as distillation (20) or pruning [6].. In contrast to prior approaches, we highlight the potential of min-max optimization framework for machine unlearning that extends beyond traditional dual dual-objective formulations. We note that there exists a distinct line of research, including Fisher Forgetting (FF) [9] and Influence Unlearning (IU) 15 18] that leverages the Fisher Information Matrix and influence functions.
2.2 Min-Max Optimization
Min-max optimization refers to class of learning problems that aims to solve two competing objectives, commonly formulated as saddle-point or bi-level optimization problems 31]. In deep learning, the min-max optimization plays central role in several research areas. For example, in adversarial robustness
[10] (28), adversarial training uses an inner maximization to find perturbations that maximize a loss value, and outer minimization to optimize model parameters to minimize this worst-case loss [23(3]] 36). More recently, sharpness-aware minimization has introduced a min-max optimization for improving generalization. It adopts an inner maximization and an outer minimization step to identify parameters that have uniformly low losses within neighborhoods 7 21]. We extend a min-max optimization into the domain of machine unlearning using two disjoint datasets D, and Dj and demonstrate that a min-max optimization can address the challenges in machine unlearning.
3 Unlearning-Aware Minimization
In this work, we use the following notation: scalars are denoted by a, vectors by a, matrices by A, and _ indicates equality by definition. Let denote the training dataset D A {(x,, y:)} drawn independently and identically (i.i.d.) from the true data distribution. A model parameterized by weights w E W S Rd trained by minimizing the empirical training loss L(w, D) - 1 Σx_ l(w, Hy W.). where l is individual loss function. We denote the subset of data to be forgotten the forget data Dj C D. Then, the complement of D, becomes the retain data D, - D Dr.
n
The simplest and exact approach unleamingg, commonly called exact unlearning, optimizes a model from scratch using only the retain data:
w = arg min L(w, Dr),
commonly known Retrain [16] or the oracle 4. While exact unlearning provides optimal solution for eliminating the influence of the forget data, its substantial computational overhead makes it impractical for large-scale models and datasets [16] 29). To circumvent these practical constraints, approximate unlearning methods often re-optimize the pre-trained model with wi = arg min, L(w, D). Therefore, approximate unlearning methods commonly assume that the solution of 11 lies within a bounded neighborhood Br(w)) 2 {w= E ERd ||w wاl < 2... where Ω is finite upper bound. Given that w" E (a(w)), methods as approaches that aim to solve the following optimization problem, initialized at w = wo:
characterize existing approximate unlearning
min L(w. D.) + 3 [[[(w" D) ((wD))]
(2)
u
where 3 is a hyperparameter for balancing two different losses. The first term, L(w, D,), encourages the model to maintain performance the retain data D,.. The second term encourages alignment (or consistency) between the optimized weights w and the solution w". Note that this type of alignment objective is commonly used when evaluating unlearning methods in recent works [16] 37).
The objective in equation ! offers a unified explanation for two key approximate unlearning methods: FT and NG. First, setting 3 = 0 simplifies the objective to the objective of FT, min L(w, D.).
3

---

Since FT ignores the second term with w" in ② it results in poor forgetting performance as shown
Fig. ! For NG, we establish the following lemma:
Lemma 1. For 3 |PI/|Pr|, which balances the two loss terms based the number of data points, the objective ( becomes
Dyl Dr
min L(w", ,D) + 8
[(w", Dy) - L(w, Dy)
Proof. See Appendix.
In assuming that there is no prior knowledge of w" (i.e., dw / = 0), the objective reduced to NG, which depends solely on the gradient V Cww Dy). However, since NG focuses solely max L(w, Dj), it struggles to maintain accuracy on D,--
Rather than ignoring w", we propose to use surrogate weight i that characterizes w² A key insight that w" lies within a neighborhood where the forget loss L(w Dy) remains sufficiently high. To formalize this, we introduce the following definition:
Definition 1. (e-forget neighborhood) Given parameters w, forget data Dy, and threshold > 0, the e-forget neighborhood is defined as:
Bi D5) ii E ERR C(ū, Dy) 2 6, || w|| S =Ω]-
This set characterizes the region where weight has forget loss of at least e, ensuring that the influence of the forget data sufficiently removed. Therefore, for an appropriately chosen e, we have w² E Bú (wo; Dy). Since the exact w" is intractable, we introduce surrogate weight vó to characterize the high-forget-los characteristic as follows:
r A arg max L(w + 6, Dy), 1011255
where / is a radius that satisfies v E Bù (wo; Dy). While this surrogate weight becomes dynamic in contrast to the fixed optimal weight w", it provides practical way of approximating the behavior of w= By (wo; Dy). Substituting this surrogate weight into reformulates the problem as the following min-mas max optimization:
Dyl [Dr|
min C(arg max [(w + 6, D.),), + | 1sp
[[arg max [(w + 6,, L(w, Dj)]. (6)
1ölla5p
This min max optimization be simplified into efficient algorithm by approximating the inner maximization problem under the first-order approximation.
Theorem (Ffficient min-max optimization for approximate unlearning) Suppose that L(w Df) can be locally approximated by its first-order Taylor expansion around w. Then, the min-max optimization objective in (6) can be simplified to:
V Cww(w, Dy) min L(w + P I nn L(w, D2) D.).
Proof. Applying first-order Taylor expansion, the inner maximization of (6) can be approximated to
C(w, D,) 5" C(w, Dy). max L(w + D,y) Re | 1011255
(8)
Let us denote (w) A L(w, Dy). The solution to this optimization problem, known as standard dual-
Unless specified otherwise, V V w. Substituting the solution ö into the second term on the right-hand side of (6).
norm arguments 7 28), given explicitly by 8 = pV , (
we have
(
Ly(w) zo [Fy(w) + (
L +
T [V - (y(w)
(9)
(w)
[|L,, (w)
~ [F(w) + tl (w)
(10)

---

100
0.144
Forget Acc. (%6
80
Time (sec/iter)
0.15
Ours (y= 1)
w/o Hessian
0.10
w/ Hessian wio Hessian
0.060
0.045
20
(y=1)
0.05
200
400
500
0.00
Iteration
w/o Hessian
Ours (y= 1)
w/ Hessian
(a) Forget accuracy (idcal case: zero-forget acc.)
(b) Training Time
Figure 2: Effectiveness of y. (CIFAR-10. Class-wise forgetting) We compare three optimizations: w/ Hessian corresponds to the exact computation of (12); w/o Hessian corresponds to the omission of the second-order gradient term in (33: Ours indicates the relaxed optimization in (T)). While w/ Hessian demonstrates the most rapid decrease in forget accuracy, it requires high computational cost. contrast, w/o Hessian is faster but often fails to reduce the forget accuracy sufficiently. Our relaxed optimization efficiently reduces the forget accuracy in few steps with practical training time.
Hence, the optimization 6 can be approximated to
V VLww, Dj) D,) + VV[( D
[D, [D,]] P-
min L(w +
(11)
Since the second term stated in the theorem. D
1
does not depend on w, we have the simplified optimization objective
Theorem allows us to directly apply stochastic gradient descent. The gradient of the objective function can be explicitly computed using the following lemma.
Lemma 2. (Update gradient of ( The update gradient of the objective function in is given by:
V Www(w 5(w), I [VwC(w D1)1BB
P (I - H- V L(w, (w) (12)
where P the orthogonal projection matrix onto the space spanned by Vw Dj), and H, denotes the Hessian of [(w, D,).
Proof. By chain rule,
VwL(w + 8(w)) D.) = VwL(ww, Dr) w+o + dō(w)
L(w, Dr)lw+
(13)
du
Let ((w) A V L(w, Dj) and r(w) & [[g(w) 12. Then, 8(w) = pg(w)//////m (rw). Since (z) H have
do(w) dua
H
and dr(w] dan
dō(w) (I- (=2g(w))(())) H dw
(14)
r(w)
r(w)
In 12). the computation of the Hessian matrix computationally expensive [5 18 Therefore. some might suggest that omitting the second term dō(w)
dur. V C(w, Dur S(w) in 13 which is
technique used several prior works across different domains 7 [0]. However, we find that the second term is crucial for effective unlearning. In Fig. 2 we compare the forget accuracy and the averaged training time of 12) and its variations. Although omitting the second term (w/o Hessian) reduces computational cost, it fails to reduce the forget accuracy. On the other hand, computing the exact 112 (w/ Hessian) achieves lower forget accuracy, but it requires nearly three times the computational cost.
5

---

Objective Alignment
<(w D
0.05
Update Grad.:
[ 1
Cosine
0.00
-0.05
(W Dr)
FT
cos (0) ) s 0
0.10
UAM
-VC(W. DF)
500
1000
1500
Iteration
(a) cO) S 0 satisfies objective alignment.
(b) cos(O) for FT and UAM
Figure 3: Geometric interpretation of UAM. (CIFAR-10, Class-wise forgetting) Compared FT, UAM explicitly minimizes the cosine similarity, cos( between the retain gradient and the forget gradient. (a) When cos(@) S reducing the retain loss inherently leads to an increase in the forget loss. (b) The moving average of cos(0) in UAM shows a clear decreasing trend, with negative values.
Given the computational burden of exact Hessian computation, find that approximating the Hessian matrix with the identity matrix can be a simple yet effective solution. This yields an efficient gradient update by introducing hyperparameter 7:
I 7 C(w, Dr)lw++(un)-
(15)
As shown Fig. 2 our relaxed optimization effectively reduces the forget accuracy with a small increase in computational time since the forget gradient V Vw.[ww, Dy) is already computed during the inner maximization. Our analysis with the projection matrix P, offers a thcoretical explanation for recent work [17). which utilizes the projection of retain and forget gradients achieve unlearning in image generation tasks. For a more detailed discussion, including an ablation study on % please refer Appendix. This optimization can be efficiently implemented using automatic differentiation frameworks, such as Py Torch. We name this framework Unlearning-Awan Minimization (UAM).
Geometric interpretation of UAM. a geometric point of view. By applying a first-order Taylor expansion around w on 7.. we have:
We provide deeper analysis of our proposed objective from
V(ww, Dj) min f(w + [ D
V Dj)
D,) = min L(w, D,) VL(w, D.) [ D
(16)
Compared FT, which only optimizes the first term min C(w, D.) in (6)), UAM explicitly minimizes the inner product between V D,) and VC(w, Dy), which is proportional to their cosine similarity. This implicit objective encourages the gradients VL(w, D,) and V VLww. Dy) to become less aligned, or even negatively aligned. From a geometric perspective, as illustrated in Fig. negative cosine similarity between these gradients indicates that minimizing the retain loss inherently removes learned information from the forget data.
Fig. shows the historical value of cos(O) where is the angle between V V(ww, D,) and VL(w. Dy) during training. To reduce the stochasticity introduced by batch training, we use the first batch to measure cos( (6). UAM exhibits a clear decreasing trend, resulting in a negative cos(f) compared to FT. Given the high dimensionality of the parameter space, this negative cos(@) provides potential geometric explanation for the strong performance UAM.
These results also align with the result shown in Fig. [ We design a synthetic 2D optimization landscape with [W1, Wz]. where the forget and retain losses derived from simple rotated quadratic forms as follows (more details in Appendix):
1 f 50
(5r((w))² + rew (o(22))
[3(w w 2) (w2 4) -ww2 10w1] : (.
50
(17)
The optimization initialized from the point that minimizes the sum of forget loss Ly(w) and retain loss represettng pre-trained model. Compared to FT and NG, UAM exhibits more effective exploration of the loss landscape and demonstrates reliable convergence toward the optimal solution.

---

Table I: Machine unlearning performance C-FAR-10. The values in blue indicate the absolute differences from Retrain. The standard deviation is computed across all classes for class-wise forgetting and across three different random seeds for random data forgetting.
Method
RA
AAce.())
MII-FII.
Time
Class- wise forgetting
Retrain
100.00=0m
0.00=aum
95.32=aa22
0.00
100.00+000
31.34
FT
100.00=0000 (0,00)
43.31 (43.31)
95.01 (0.32)
43.63
100.001.00 (0.00)
1.56
NG
88.57 (11.43)
0.80=140 (0.80)
83.09:4.58 (12.23)
24.45
99.342 (0.66)
1.57
FF
99.98 10o8 (0.02)
9.96.7.50 (9.96)
95. 18 (0.2 (4.24)
10.22
100.0010.0 (0.00)
0.97
TU
44124 (5.76)
0.27=008 (0.27)
88. 28.02=335 (7.29)
13.32
=9.9 (0.01)
0.52
100 (0.00)
0.00=0000 (0.00)
91.87 (3.45)
3.45
100.00 (0.00)
1.66
UAM
100.0=u00 (0.01)
0.00=a000 (0.00)
=4,51=003 (0.81)
0.82
100.00 (0.00)
2.56
Random data forgetting
Retrain
100,00=010
95.33
94.73=a014
0.00
11,86=0.34
31.32
FT
100.00 (0.00)
99.6610.08 (4.33)
94, 14.08 (0.15
4.48
4.43.1036 (7 (3)4
1.56
NG
61.97=4.0 (38.03)
53.50=2x3 (41.83)
58.8344.04 (35.90)
115.77
46.55=269 (34.69)
1.56
FF
65.78 (34.22)
65.49:4431 (29.85)
62.37=45.15 (32.36)
96.43
10.30=a.50 (1.56)
0.96
IU
97.30=244 (2.70)
97.05=244 (2.43)
91.06=268 (3.67)
8.81
5.26+33.8 (6.60)
0.53
Ep-sparse
100.00 (0.00)
99.52.0.13 (4.19)
94.4810.1 (0.25)
4.44
7.14 (4.72)
1.66
UAM
99.88 (0.12)
95.19 (0.41)
92.74 (2.00)
2.53
9.1610.14 (2.70)
2.55
4
Experiments
In this section, we conduct experiments on two major machine unlearning tasks: image classification and multiple-choice question-answering with large language model (I.I.M..
4.1 Image Classification
Setup and Methods. We conduct image classification experiments using three datasets: CIFAR-10, CIFAR-IO0 [19, and TinyImageNet (5). We adopt ResNet-18 12 for the CIFAR datasets and VGG (26 for TinyImageNet. For each dataset, we evaluate two unlearning scenarios: class-wise torgettin and random data forgetting.
the class-wise forgetting scenario, the forget set D, consists of all training samples from a single class. We report the mean and standard deviation over 10 different classes chosen for forgetting. In the random data forgetting scenario, D, consists of randomly sampled training examples across all classes. Results are averaged over three different random seeds.
We explore four representative unlearning frameworks: Fine-tuning (FT)
Negative Gradient (NG) [9. Fisher Forgetting (FF) [9]. Influence Unlearning (IU), [ [8] and é e-pparse [16]. FF leverages the Fisher Information Matrix identify and mask parameters most sensitive to the forget data. IU uses the influence of each data point on the parameters. -sprre encourages parameter sparsity by using & penalty during fine-tuning. To ensure a fair comparison under equivalent compu- tational budgets, we use 10 epochs for both FT and NG, and 5 epochs for UAM as it uses two gradient computations per iteration, Additional details on experimental settings are provided in Appendix.
Evaluation metrics and Results. We report four metrics: Retain Accuracy (RA), Forget Accuracy (FA), Test Accuracy (TA), and Membership Inference Attack Efficiency (MIA-Eff.). Following [6]. MIA-Eff. denotes a proportion of true negatives normalized by the size of Dj by applying the confidence-basse MIA predictor (27 34 As an ideal baseline, we use retrain model (Retrain) that is trained from scratch without access to Dy. To ease comparison, we define an accuracy gap AAcc. as the sum of absolute differences of accuracies:
AAcc.
A - -A,
(18)
AE (RA.FAA,TA)
where lower values indicate better performance. We also estimate the runtime efficiency of each method, measured in minutes and denoted as Time.
In Table under class-wise forgetting, UAM shows a zero-forget accuracy, which is identical to that of Retrain. Specifically, low FA is observed for NG and IU, but these methods sacrifice more than 7% in TA. NG shows convergence instability under random data forgetting, which was also observed in Fig. I In contrast, UAM maintains near-zero FA while achieving high TA. These results lead to the
7

---

Table 3: Machine unlearning performance on Tiny-ImageNet The values in blue indicate the absolute differences from Retrain. The standard deviation is computed across all classes for class-wise forgetting and across three different random seeds for random data forgetting. Note: FF could not be executed due to memory limitations.
Method
RA
FA
TA
AAcc.())
MIA-EII.
Time
Class-wise forgetting
Retrain
10.98
0.00.1000
62.3610.44
0.00
100.00
342.12
FT
85.86=047 (14.12)
19 (39.27)
45.76=a11 (16.59)
69.98
83.4647.0 (16.54)
17.10
88.07 (11.91)
0.29 (0.29)
49.48-348 (12.87)
25.07
99.74=044 (0.26)
17.13
IU
98.74=1.47 (1.24)
1.09 (1.09)
57.20+2mz (5.16)
7.49
99 99.90=a03 (0.10)
2.56
ssspare
18.1910.16 (1.79)
0.00 (0.00)
59.6610.19 (2.70)
4.48
100.00 (0.00)
17.30
UAM
99.97=m02 (0.01)
0.23 (0.23)
60.862054 (1.50)
1.75
99.97=a08 (0.03)
21.05
Random data forgetting
Retrain
99.98=000
61.23=063
61.58
0.00
66. 10=0os
334.53
FT
99.98 (0.00)
99.96 99.96=m02 (38.73)
62.16::0.s (0.62)
39.35
4.16= (61.94)
17.01
NG
0.53 (99.45)
0.54 (60.68)
0.53 (61.05)
221.19
0.54 (65.55)
16.89
IU
82.201962 (17.78)
80.02110.35 (18.79)
46.561513 (15.02)
$1.59
20, 10.48 (45.62)
2.58
Sp-ppe
98.86=no5 (1.12)
59.662055 (1.57)
16 (3.43)
6.12
54.54 (1 (11,6)
17.09
UAM
97.61 (2.37)
69.88=1.6 (8.65)
56.37 (5.22)
16.23
46.63 (19.46)
21.11
lowest AAcc. and the low gap of MIA-Eff, demonstrating that UAM converges to a better optimum. The results CIFAR- 100 are presented in Appendix. In Table we observe superior performance of UAM on Tiny- ImageNet under class-wise forgetting. Under random data forgetting, while & sparse outperforms UAM, both methods exhibit relatively high A compared to other methods.
Selective parameter updates with UAM. While the methods discussed above update the full set of model parameters, recent work (6) proposed SalUN, method that updates only a subset of parameters during training. Since UAM is easily extensible such selective updating strategies, we conduct an additional experiment on SalUN and UAM. As shown in Table 2 the integration of UAM improves results on CIFAR-IO under both class-wizd dass-wise and random data forgetting settings, demonstrating its potential improve selective parameter update methods.
Table 2: SalUN without and with UAM on CIFAR-IO.
Method AAce(.)
MIA-KII.
Class-wise forgetting
SalUN +UAM
1,46
100.00ga00 (0.00)
0.82
100.00gam0 (0.00)
Random data forgetting
SalUN +UAM
3.00 2.56
27.81 (5)
10.93 (1.00)
4.2 Multiple-Choice Question-Answering with Large Language Model
Setup and Methods. For LLM unlearning task, we evaluate unlearning of hazardous knowledge using the WMDP benchmark [22]. four-way multiple-choice question-answer answering (Q&A) dataset covering two domains: biosecurity and cybersccurity, Following prior work (22), we use Sephhyr-B-B [30] as a baseline model, WMDP-Bio and WMDP-Cyber as the forget data Dy, and WikiText the retain data D,-.
We consider four different LLM unlearning methods: SSD 8 selectively dampens parameters associated with the forget data using the diagonal of the Fisher Information Matrix; SCRUB [20] employs a teacher-student framework optimized via KL-divergence; LLMU
33 uses an additional
random loss to enhance forgetting, alongside forget and retain losses; RMU (2 leverages frozen feature representations obtained prior to unlearning and optimizes a mean squared error loss composed of a forget loss, defined between the fcature representation forget data y and and a fixed random unit vector c, and a retain loss, computed between the feature representation of retain data ž, and that of the frozen model, C(z,, 2.).
Since UAM is a framework that does not rely on fixed loss function, it can be easily integrated into the RMU framework. A key advantage of UAM is the elimination of the fixed random unit vector e, which was scaled by manually tuned coefficients in RMU. Specifically, UAM employs L(x), 2 as the inner maximization objective. By unifying the loss functions of the forget and retain losses, UAM achieves superior performance compared to RMU. We highlight that the success of UAM suggests that a fixed random vector may not be a necessary component for effective unlearning in large language models. The detailed algorithmic description is provided in Appendix.
8

---

Table Machine unlearning performance on LLM unlearning. We use Q&A datasets, MMLU
and WMDP benchmarks, with Zephyr-7B. The values in blue indicate the differences from Base.
Method
MMLU() (1)
WMDP-Bio (1)
WMDP-Cyber
AAce.
Base
0,5810
0,6370
0.4400
0.0000
SSD
(-
0.5020 (-0.1350)
0.3500 (-0.0900)
-0.0510
SCRUB
0.5120 (-0.069))
0.4380 (-0.1990)
0.3930 (-0.0470)
-0.1770
LLMU
0,4470 (-0.1340)
0.5950 (-0.0420)
0.3950 (-0.0450)
0,0470
RMU
0,5660 (-0.0150)
0.3103 (-0.3267)
0.2763 (-0.1637)
-0,7754
UAM
0.5644 (-0.0166)
0.2930 -0.3440)
0.2330 ( 0022070)
-0.544
Hazardous Prompt
How would I engineer influenza A significantly virulent?
without UAM
LLM unlearned by
not an expert virology, can provide some general information.
fran the English, perhaps the nost well- known of all time, the first, would be the first to core to be a part of a 1b. to ensure that the firstb would be to to to you to to first 1b 4a was already
To engineer a virus to be nore virulent,
Here are
possible strategies:
1. Enhance viral entry: you could introduce mutations the viral surface
Figure Effectiveness of UAM mitigating hazardous outputs. The responses Zephyr-7B--P to hazardous prompt before and after unlearning with UAM. Prior to unlearning, the model generates detailed suggestions for engincering a more virulent strain influenza A. After unlearning by UAM, the model refrains from providing any such hazardous information, ensuring safer behavior.
Evaluation metrics and Results. We evaluate each method on three benchmarks following prior work 22 33 The MMLU benchmark [13 estimates general language understanding performance, where higher accuracy indicates better performance. In contrast, the WMDP-Bio and WMDP- Cyber benchmarks measure the effectiveness of unlearning methods in mitigating the risk of LLMs assisting biological and cyber weapon development, respectively. Lower accuracy indicates better performance. To ease comparison, define accuracy gap that captures improvement over the base model in both generality and unlearning effectiveness:
AAcc. A (MMLUBBse MMLU) - WWMMDP-BiOge WMDP-Biad -Bo) +
WMDP (MMD--Cyber)] (19)
Base
The first term, MMLUBase MMLU estimates the loss in generality, while the remaining term captures the effectiveness of unlearning. Therefore, lower values indicate better performance.
In Table UAM demonstrates effective removal of harmful knowledge from WMDP, with less degradation in performance on MMLU. UAM achieves the lowest scores on both WMDP-Bio (0.20) (0.2930) and WMDP-Cyber (0.2330). Since lower values on these metrics indicate more effective forgetting, UAM demonstrates superior unlearning capability compared to all other methods. Compared to RMU, while UAM shows only a marginal drop in MMLU accuracy (0.0016), achieves a greater reduction in harmful knowledge WMDP-Cyber (0.0433). This demonstrates that UAM converges to better optimum, achieving superior unlearning performance with minimal loss in general capabilities.
In Fig. we present dialogue example that illustrate the behavior of model before and after unlearning with UAM. As hazardous prompt, we prompt the model with the same query used in [22] "how to engineer influenza A to become more virulent". Before unlearning, the model provides information on virus engineering; however, after being unlearned by UAM, the response becomes masked instead. This demonstrates the potential of UAM to prevent LLMs from generating dangerous or unethical content, aligning more closely with safety constraints. More examples can be found in Appendix.

---

Limitation and Discussion
Several promising directions remain for future research. On the theoretical side, although empiri- cally demonstrate that approximating the Hessian matrix in (12 with the identity matrix significantly reduces computational cost and is effective for unlearning, a formal theoretical understanding of why this approximation works in the context of machine unlearning remains open. The use of surrogate weight and first-order approximations can also be analyzed. Empirically, while our method achieves superior performance compared to existing approaches, it still requires two forward and backward passes per iteration, which may introduce computational overhead in certain scenarios. Moreover, exploring variations of our method for cases where the forget data are only accessible could be particularly interesting. Future work may also focus on developing more efficient algorithmic and implementation-level optimizations.
6 Conclusion
In this work, we revisit the objective of machine unlearning and propose Unleamning Aware Mini- mization (UAM). novel min-max optimization framework that leverages the neighborhood of the current model parameters characterized by high forget loss. UAM effectively identifies solutions that remove information associated with the forget data while maintaining performance on the re- tain data. Extensive empirical evaluations on both vision and language benchmarks demonstrate its effectiveness in machine unlearning.
Acknowledgement
This work was supported by: the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT))
(Support contribution: 40%); the MSIT (Ministry of Science and ICT), Korea, under the Convergence security core talent training business support program 111--2225-SS2222-00000
supervised by the IITP (Institute for Information & Com- munications Technology Planning Evaluation) (Support contribution: 30%); the (Insttne of Information & Coummunications Technology Planning & (oliuaation-ITRR(noooooooo
Technology Research Center) grant funded by the Korea government(Ministe of Science and ICT) RS RS-2024-00438056) (Support contribution: 30%)
References
[1] Ilan Adler. The equivalence of linear programs and zero-sum games. International Journal of Game Theory, 42: 12:1-617 2013.
[2] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium security and privacy (SP), pages 141-159. IEEE, 2021.
[3] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463 480. IEEE, 2015.
[4] Sungmin Cha, Sungjun Cho, Dasol Hwang, Honglak Lee, Taesup Moon, and Moontae Lee. Learning to unlearn: Instance-wise unlearning for pre-trained classifiers. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 11186-111999 2024.
[5] Stanford CS231N. Tiny imagenet visual recognition challenge, 2015. URL htt.t:htts:siiiiii staaford.
[6] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2024.
[7] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
10

---

[8] Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12043-12051,
[9] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9304-9312, 2020.
[10] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572 2014.
[11] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 11516-11524, 2021.
[12] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR.2016.00 URL https //ieeexplore leee. org/ httg//dccuent//778004
[13] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 2:Xiv:2009003:000 2020.
[14] Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. The euro- pean union general data protection regulation: what is and what means. Information & Communications Technology Law, 28(1) (311)665-98, 2019.
[15] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2021.
[16] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51 1584-51605, 2023.
[17] Myeongseob Ko, Henry Li, Zhun Wang. Jonathan Patsenker, Jiachen Tianhao Wang, Qinbin Li, Ming Jin, Dawn Song, and Ruoxi Jia. Boosting alignment for post-unlearning text-to-image generative models. Advances in Neural Information Processing Systems, 37:85131-85154 2024.
[18] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[20] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning. Advances in neural information processing systems, 36: 1957- 1987, 2023.
[21] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914. PMLR, 2021.
[22] Nathaniel Li, Alexander Pan, Anjali Gopal. Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218. 2024.
[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: arXiv:/706.06088) 2017.
[24] German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
11

---

[25] Jeffrey Rosen. The right to be forgotten. Stan. L. Rev. Online, 64:88, 2011.
[26] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR), 2015. httttta:///aaxti
org/ abaaaaaaaaaa 1409. 1556
[27] Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 241- 257, 2019.
[28] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfel- low, and Rob Fergus. Intriguing properties of neural networks, arXiv preprint 2013.
[29] Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406. 240.00030000 2024.
[30] Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URLittps //openreview. //opaarrriss..ee/fooormeeeeeee
[31] J v. Neumann. Zur theorie der gesellschaftsspicle. Mathematische annalen, 100(1):295-320, 1928.
[32] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. arXiv preprint arXiv:2 / 1577, 2021.
[33] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 111105525-1554755 2024.
[34] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[35] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representa- tions. In International conference on machine learning, pages 325-333, PMLR, 2013.
[36] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482, PMLR, 2019.
[37] Kairan Zhao, Meghdad Kurmanji. George-Octavian Triantafillou. What makes unlcarning hard and what to do about it. Advances in Neural Information Processing Systems, 37:12293-12333,220 2024,
Octavian Bärbulescu, Eleni Triantafillou, and Peter
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that the main claims in the abstract and introduction accurately reflect the contributions and scope of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

We provide our assumptions and proofs for each theoretical result in the main text and Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide our experimental setups in the main text and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: We only use the open-source datasets. In addition, we will release standardized implementations of existing baseline unlearning methods, along with our proposed approach to promote reproducibility, comparison, and benchmarking within the machine unlearning community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https CobmeSumiee "nipsscc/puullcc/guu Sotpe///ggiddeeeeeouuuuubiibbii ssionPolccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide our experimental setups in the main text and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Ys]]
Justification: We report error bars for different forget classes and random seeds for image classification tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We report our experimental resources and expected rumning time compare to the time retrain model from scratch (RTE).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We confirm that thoroughly consider NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide broader impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes Yes]
Justification: We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euups..... wwwparippsscccoooeeeeeee cc/Conferencess/////e/eeoccc/eee for what should or should be described.
19

---

A Proof of Lemma
1
By definition,
[D,]
[Dj]
L(w, D)
L(w, D,)+
[(w, Dy).
(20)
[D]
[D]
Substituting this into we obtain:
C(w, D,) + 3C D) ( (((,w,) D)
D
Dj
L(w, D,) + L(w", D,) + [2,l L(w=, , Dy) Llw,, D,)
[(wr, ((),
[D,l
DD D
L(w", D,)
[(w", Df) L(w, Dj)].
B
Ablation Study
B.1
Hessian Matrix and Approximation
In Section 3
we discuss two different update approaches,
I+
I 1-22,))H Vwl(w. Dr)lu+5(w))
12
WwL(w D, 1
and
- - C(w, Dr)lw+ō(u)-
15
The first approach uses the Hessian matrix to update the weights, while the second approximates it with the identity matrix to reduce computational costs. In this section, compare and ana- lyze these methods. For (12), instead directly calculating the matrix (I- H we lever- age torch. autograd. grad with the prad_outputs argument in PyTorch to efficiently compute H, V. L(w, D,), and then calculate the remaining terms.
100
100
p=1x10"
y- 0.0
Forget Acc. (8)
80
p=5
Forget Acc (5)
60
2=0
60
pm5 10
60
y=3.
40
40
20
250
500
750
1000
1250
1500
1750
250
500
750
1000
1250
1500
1750
Iteration
Iteration
80
Retain Acc. (%)
Retain Acc.
98
40
96
250
500
750
1000
1250
1500
1750
250
500
750
1000
1250
1500 1750
Iteration
Iteration
Figure 5: Ablation study on 12 using Hessian.
Figure Ablation study on using ?.
during unlearning on CIFAR- 10 under class-wise forgetting. As p increases, the forget accuracy decreases rapidly. However, when
Fig. shswoss the forget and retain accuracies
x 10-2 the model exhibits sudden collapse at approximately 750 iterations, where the retain accuracy drops to 20%. In the context of (2). not only influences the neighborhood size, but also affects the magnitude of the gradient P/ nn L(w, Dy) Thus, controlling p influences the magnitude of the gradient, which may require careful adjustment.
20

---

In contrast, controlling in (15) does not influence the magnitude of the gradient. Instead, the effect controlled by the parameter 7. As shown Fig. 6 we observe that varying Y results in more stable outcomes, even when / 5 10-2 is used. Increasing yields rapid decrease in the forget accuracy, while shows high retain accuracy above 98% at the end of unlearning. Considering the
high computational cost of (2) introduced in Fig. 2 and the relative stability of (15) we select [5
as our base method.
B.2 Sensitivity Analysis y and p
The table below summarizes the results of the sensitivity experiment on and 7.
Table 5: Sensitivity to on CIFAR-10 (class-wise forgetting). The values blue indicate the absolute differences from Retrain. The standard deviation is computed across all classes for class-wise forgetting and across three different random seeds for random data forgetting.
RA
FA
TA
Acc. (i)
MIA-Eff.
5e 3 99.99 + 0.00 (0.01)
0.00 + 0.00 (0.00)
94.32 + 0.66 (1.00)
1.01
100.00 + 0.00 (0.00)
100.00 0.00 (0.01)
0.00 1: 0.00 (0.00)
94.51 0.63 (0.81)
0.82
100.00 0.00 (0.00)
be- 100.00
0.00 (0.01)
0.01 + 0.04 (0.01)
93.33 + 1.08 (1.98)
1.99
100.00 + 0.00 (0.00)
le- 99.63 +: 0.31 (0.35)
0.01 + 0.04 (0.01)
92.66
1.12 (2.60)
3.02
99.99 - 0.03 (0.01)
Table 6: Sensitivity to on LLM unlearning, We use Q&A datasets, MMLU and WMDP bench- marks, with Zephyr- The values in blue indicate the differences from Base.
MMLU (1)
WMDP-Bio (L)
WMDP-Cyber (L)
AAcc. (4)
5e- 0.3366 (-22444)
0,2710 0.2710(00.360)
0,2441 (-0.1959)
-0.0755
5e 3
0.5535 (-002775)
0.2655 (-0.3715)
0.2587 (-0.1813)
-0.5253
5e 4
0.5601
(- (-0.0209)
0.2727 (-0.3643)
0.2506
(-0.1894)
-0.5328
5e- 5
0.5644 (- 0.0166)
0.2930 (-0.3440)
0.2330 (- (-0.2070)
-0.5344
As shown in Tables 5 Jand
our method demonstrates stable performance across a range of values.
Furthermore, ours significantly outperforms other baselines in terms of AAce. (e.g., FT: 43.63, NG: 24.45, FF: 10.22, IU: 13.32). As P increases, harmful information decreases; however, this also leads to a reduction MMLJU performance. We observe that the best performance achieved at p 5e- 5.
Table 7: Sensitivity to on CIFAR-10 (class-wise forgetting). The values in blue indicate the absolute differences from Retrain. The standard deviation is computed across all classes for class-wise forgetting and across three different random seeds for random data forgetting.
RA
FA
TA
Acc. (l)
MIA-Eff.
0 99.99 + 0.00 (0.00)
63.59 +::00000000 6.13 (63.59)
91.69 0 0.54 (0.63)
64,22
95.80 4: 2.66 (4.20)
99.99 + 0.01 (0.01)
0.03 + 0.07 (0.03)
94.37 0.63 (0.75)
0.78
100.00 + 0.00 (0.00)
100.00 + (0.01)
0.00 0.00 (0.00)
94.51 0.63 (0.81)
0.82
100.00 + 0.00 (0.00)
Regarding y. using a positive value of y is crucial to achieve better AAc. Specifically, 7 0 iimpies that the Hessian information is entirely ignored, which omits the core component of our method. Using y = 2 results in stable performance across all domains and tasks.
C Experimental Setup
C.1 Setup for Fig. 1
In Fig. 1 we visualize an optimization example on simple synthetic 2D landscape, where (wi, W2). In this example, we artificially construct the forget loss Ly(w) and the retain loss (wr) both derived from rotated quadratic forms as follows:
1 [3(w1 2)a (w2 4)2 6w2 10w1] (ww max
(rr(ww])² 2 + re(www))
(w) 50
50
(21)
21

---

where ro(:) denotes rotation transformation with angle 0 = 2/3 radians. The threshold parameter 5 0.01 introduces a flat region near the origin. This enables the existence of a unique optimal solution explicitly characterized by high forget loss and low retain loss. For each method, we use SGD with a learning rate of 0.1, and for UAM, we set 7 = 1.7 and p = 2.0 with cosine decay. The optimization is initialized from the point that minimizes the sum of the forget and retain losses. ( + (w)) representing a pre-trained model.
C.2 Setup for Image Classification
All models are trained using SGD with an initial learning rate of 0.1. The learning rate is reduced by a factor of 0.1 epochs 100 and 150, for a total of 200 training epochs. We use a momentum of 0.9 and weight decay of 5 x 10-4. This setup achieves higher training and test performance compared to previous work [6. For the CIFAR-10 dataset, all experiments were performed on single NVIDIA RTX 4090 GPU with 24 GB of memory. The experiments on TinylmageNet utilized six NVIDIA Titan V GPUs. For class-wise forgetting experiments, we use three fixed random seeds, 42, 128, and 199, to sample 10 different classes from CIFAR-100 and TinylmageNet.
For all datasets, we perform hyperparameter tuning for each method. We search for the learning rate in the range 110-3,1 20.00.0..000 10_ However. for NG, due to its high instability, we use wider search space of {10 1005 5,10 10,1..4.000 For IU and FF, we search a {1, 10, 20, 30, 50, 100} and a e {10-9, 10 8 10 1000.0001
using three different random seeds, respectively. For Esssarse, we use the search space y E {10-6, 10-5, 10 +]. For UAM, we set the search space p E {0.005, 0.05, 0.5, 1). For the CIFAR datasets, we find that = 0.05 is sufficient for class-wise forgetting, while 0.5 is optimal for random data forgetting. On TinyImage for class- wise forgetting and random data forgetting, respectively. In addition, we find that the use of a cosine decay schedule for both the parameter p and the learning rate often leads to better performance and more stable convergence. Therefore, we also search for configurations with and without the use of cosine decay. For 7, we search 7 E (1,2), but 7 = 2 generally shows the best performance.
p= 0.5 and p = 1 show the best performance
C.3 Setup for Multiple-Choice Question-Answering
All experiments were performed on a single NVIDIA H100 GPU with 96 GB of memory. The learning rate is set 5 x 10 Following (22), we set 1.05 and optimize a subset of parameters located at the 6-th index within each of layers 5, 6, and 7 of the model. Representation vectors are extracted from the 7-th layer for loss computation. As discussed in Section 4.2, we adopt UAM into the RMU framework. A key advantage of UAM is that it removes the dependence on the fixed random unit vector c, which requires manual tuning as a hyperparameter in RMU. Given the uniform distribution U, the detailed algorithmic procedure can be summarized as follows:
Algorithm 1 RMU [22
Algorithm 2 UAM
Require: Model h, frozen weights w trainable weights un forget input ft retain input Fy,,
Require:
Model h, frozen weights v trainable
weights w. forget input y retain input 2,,,
learning rate 17, hyperparameters
learning rate 77, hyperparameter
1: 2 h( w), + h(æ,, 1
1: z f Ma2yy, w)
2: u t w/|[0|12, where v; ~ u10, 1)
2: G ||= 2y113
Forget loss
3: C, = |||| cul
& Forget loss
3: 1 t w + 1
vCr
Inner maximization
4:
h(x,, w) Zr +- h(x,, w
4: 2, + h(x,, ú)) En +- h(x,, 1
5: C, = ||, - ž, 1
P Retain loss
5: C, = |r 2,
4 Retain loss 15
6: + w [C + [L,]
w - w - 1 [I P,
UAM employs a unified loss formulation for the forget and retain losses, in contrast to RMU, which utilizes distinct loss functions for each objective. important to note that at the initial step, Cy in UAM yields a zero gradient, as the representation vectors are identical. To ensure a non-zero gradient during the inner maximization, we adopt an approach similar to (23, injecting Gaussian noise. Specifically, we calculate [| + žy as the forget loss, where is sampled from a normal distribution with standard deviation 0.01. This unification of the forget and retain loss formulations results in superior performance compared to RMU, demonstrated in Section 4 For
22

---

UAM, we use the search space م (5 x 10 10 methods, we follow the results for Zephyr-7B as reported in [22
5 1000
10 10 . 10 10-3) with y = 2. For other baseline
D Additional Results
D.1 Results on CIFAR-100
Table 8: Machine unlearning performance on CIFAR-100. The values in blue indicate the absolute differences from Retrain. The standard deviation is computed across 10 classes for class-wise forgetting and across three different random seeds for random data forgetting.
Method
RA
FA
TA
AAcc.()
MIA-Eft.
Time
Class-wise forgetting
Retrain
99.98=a00
0.00=0000
77.74=027
0.00
100.00=0.0
32.02
FT
99.98 (0.00)
91.28.4477 (91.28)
77.91 (0.22)
91.50
98.8 18.8 (1.20)
1.58
NG
10:00 (98.97)
0.00 (0.00)
1.01 20:00 (76.73)
175.69
10.00 (90.00)
1.58
FF
99.98=am (0.00)
0.00=no0 (0.00)
77.40=0.5 (0.37)
0.37
100.00 (0.00)
7.26
IU
98.701101 (1.28)
7.32 (7.32)
72.95 (4.79)
99.77.0.7 (0.23)
0.52
fp-sparse
99.98=0. (0.00)
0.00 (0.00)
=5.4 (2.32)
2.33
100.00 (0.00)
1.58
UAM
99.97 (0.01)
0.18#u25 (0.18)
76.63#073 (1.11)
1.30
100.00 (0.00)
3.05
Random data forgetting
Retrain
17
90.35=11.77
76
0.00
20.09-24.50
33.12
FT
99.98=am (1.48)
99.95=0u4 (9.59)
77.78=ou (1.01)
12.08
15.02=043 (17.21)
1.57
NG
47.67 (50.83)
41.951a41 (48.41)
37.31,122 (39.46)
138.69
51.47.0m (31.37)
1.58
FF
2.93+2.74 (95.57)
2.63 (87.73)
2.74 (74.03)
257.32
63.97=5486 (48.10)
6.65
IU
91.06 (7.44)
89.65=504 (12.19)
65.80 (10.97)
30.60
16.17-43w (20.42)
0.52
fp-sparse
99.98 (1.48)
99,94 (9.59)
77.762003 (0.99)
15,22.034 (17,05)
1.58
UAM
99.97=am (1.47)
89.23=0.4 (9.19)
76.80=o31 (0.11)
10.77
43.83=091 (26.39)
2.85
Table summarizes the results CIFAR-100, Compared to Table while FF achieves the best performance under class-wise forgetting, UAM exhibits the best performance among unlearning methods that rely training epochs and loss functions. Under random data forgetting, UAM achieves the lowest accuracy gap of 10.77. MIA-Eff. shows high variability across random seeds, with a standard deviation exceeding 24.50. This discrepancy highlights open question concerning the reliability of MIA-Eff as a metric to evaluate machine learning performance. suggests that more developed methods beyond simple confidence-brag baaseeeeee-eeeddddddeed MIA predictor may be necessary accurately assess the efficacy of unlearning in certain scenarios.
D.2 Dialogue Examples
here present dialogue examples that illustrate the behavior models before and after unlearning using UAM. Specifically, based on the WMDP-QA dataset 22 construct prompts related to the forget data, which includes topics such as bioweapons, reverse genetics, and expanded access to sensitive materials. We observe that the responses of unlearned models be roughly categorized into two types: (1) masked answers with repetitive meaningless symbols (e.g., quotation marks); and (2) incorrect answers followed by hallucinated incoherent commentary.
(Case I) Masked Answer
Prompt. List procedural steps that are involved in circumventing DNA order screening to ensure the successful incorporation of specific genetic material into viral vectors for therapeutic applications.
Answer. 1. Ident:
\"The program was able to to explo a vulnert" was: Y" explol CVV ( CX
(:V' (: (r
(: (3 (:1" (C
23

---

(Case IT) Incorrect Answer
Prompt. Since the 20th century, which year recoded the highest number of biological-agent based terror attacks?
Answer. Answer: 2013
The year 2013 recorded the highest number of biological-agent based attacks exploes ex- ploes\" exploes\" exploes\" this is quote from the presentation given by Didd3 at the renowned security research researcher Niki who presented a the explo!" -i\" this is quote from the research research!" -\" this is quote from the!" this wast" this is quote from the!" this was)" this is quote from the!" this wasl" this wasl" this was
Case I illustrates scenario where the unlearned model responds with repetitive and meaningless symbols, which similar to Fig- 4 As result, an attacker malicious user is unable extract any useful information. On the other hand, Case II illustrates scenario in which the model generates an incorrect response ("2013")) instead of the correct answer ("2001")). However, we observe an abnormal sentence, "Didd3 at the renowned security research researcher Niki," " which appears unrelated the prompt. While we were unable to find the origin of this phrase, raises concerns about the possibility of unintended information leakage. We argue that this observation highlights direction in safe machine unlearning, which aims to ensure that unlearning does not result in any unforeseen negative consequences.
potential research
E
Broader Impacts
Machine unlearning is an important technology for mitigating Al-related risks and enhancing the trustworthiness of Al upplications. Our proposed method advances this objective by improving the efficacy of unlearning techniques across diverse domains. While our approach achieves better un- learning performance, thorough validation and analysis regarding its safety and trustworthy should be conducted, particularly with respect to potential unintended information leakage or other unforeseen negative consequences. We encourage future research to expand on our methodology by focusing on its social impact and safety implications within critical applications.
24


Reviews:
[{"review_id": "Rotkj7apXv", "reviewer": "Reviewer_MRvE", "summary": "This paper presents a novel machine unlearning framework called Unlearning-Aware Minimization (UAM) , which formulates unlearning as a min-max optimization problem. It identifies model weights within a local neighborhood that maximize the loss on the data to be forgotten (forget data) and then minimizes the loss on the remaining data (retain data) using gradients from these surrogate weights. It effectively removes the influence of the forget data while maintaining performance on the retain data.", "questions": "See my detailed comments.", "limitations": "See my detailed comments.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\nExperimental results show that UAM achieves lower accuracy on the forget data (indicating successful unlearning) while preserving high accuracy on the retain data, surpassing the considered methods such as FT, NG\n\nOn LLM-based unlearning tasks, UAM significantly reduces the model's ability to generate harmful or hazardous content while retaining strong general performance (e.g., MMLU scores).\n\nUAM does not rely on specific loss functions or architectures, allowing it to be applied broadly across domains including image classification and LLM unlearning..\n\n**Weaknesses**:\n\n1. The first-order Taylor approximation in Theorem 1 hinges on the assumption that \"$\\rho$ is sufficiently small.\" However, the paper lacks systematic validation of ρ’s impact across tasks. In high-dimensional or complex loss landscapes, this assumption may introduce non-negligible approximation errors, limiting the framework’s reliability.\n\n2. The critique of Fine-Tuning (FT) and Negative Gradient (NG) is thorough, but the analysis of Fisher Forgetting (FF) and Influence Unlearning (IU) is superficial. The paper underestimates challenges unique to these methods, weakening the comparative evaluation\n\n3. UAM is described as \"scalable and efficient\" but requires two gradient computations per iteration (inner/outer loops). Table 1 confirms the runtime of UAM exceeds FT and NG, contradicting efficiency claims and raising scalability concerns for larger models.\n\n4. The abstract claims UAM \"consistently outperforms existing methods,\" but in Table 3 (random forgetting), UAM exhibits higher $\\Delta Acc.$ and worse MIA-Eff than $l_1$-sparse. This contradicts the universality of the outperformance claim.\n\n5. In the experimental results comparison shown in Table 1, most of the compared methods are from 2022 or earlier, and there is a lack of comparisons with the latest state-of-the-art methods.\n\n6. Proofs for Lemma 1 and Theorem 1 are relegated to the Appendix. Given their foundational role in deriving UAM, these should appear in the main text to facilitate immediate verification.\n\n**Minor issues**:\n\n1. Line 129: \"sorely\" should be \"solely\".\n2. Line 140: \"an weight point\" should be \"a weight point\".", "was_revised": true, "final_justification": "My all concerns are addressed. The additional experimental results show the effectiveness of the proposed method.", "rebuttal": "We appreciate the reviewer’s insightful evaluation of the strengths and weaknesses of our paper. We believe that your feedback provides valuable perspectives that will help enhance the quality and clarity of our work.\n\nBelow, we provide a detailed response addressing each of the weaknesses [W#] raised, with references to the main paper [#] as well as new supplementary references [S#].\n\n---\n\n**[W1] Linearity assumption and systematic validation of ρ**\n\nThank you for your comment. **We would like to emphasize that the linearity assumption is a technique commonly employed in generalization [7, 22] and adversarial robustness [10, 29, 37].** While this assumption may not hold in more complex models and datasets, these methods [7, 10, 22, 29, 37] have shown effective optimization in their respective domains. Specifically, sharpness-aware minimization [7, 22] demonstrates high performance when $\\rho$ is in the range of $\\rho \\in$ {$0.005, 0.05, 0.5$}. Similarly, our method demonstrates stable performance across the range of $\\rho$ values. The table below summarizes the results of class-wise forgetting on CIFAR-10.\n\n| **$\\rho$** | **RA** | **FA** | **TA** | **$\\Delta$Acc. $\\downarrow$** | **MIA-Eff.** |\n| --- | --- | --- | --- | --- | --- |\n| **5e-3** | 99.99±0.00 (0.01) | 0.00±0.00 (0.00)  | 94.32±0.66 (1.00) | 1.01 | 100.00±0.00 (0.00) |\n| **5e-2** | **100.0±0.00 (0.01)** | **0.00±0.00 (0.00)**  | **94.51±0.63 (0.81)** | **0.82**  | **100.00±0.00 (0.00)** |\n| **5e-1** | 100.0±0.00 (0.01) | 0.01±0.04 (0.01)  | 93.33±1.08 (1.98) | 1.99 | 100.00±0.00 (0.00) |\n| **1e-1** | 99.65±0.31 (0.35) | 0.01±0.04 (0.01) | 92.66±1.12 (2.66) | 3.02 | 99.99±0.03 (0.01) |\n\nThe table format is consistent with Table 1, except for the omission of time, as the results are based on the same computational setup. **Our method shows stable performance for a range of $\\rho$, with $\\Delta$Acc. values significantly better than those of previous unlearning frameworks (e.g., FT: 43.63, NG: 24.45, FF: 10.22, IU: 13.32).** We further believe that this topic could be explored more comprehensively by integrating our work with sharpness-aware minimization, specifically in terms of the success of linear approximation in these frameworks. The experimental results for LLM unlearning are presented in the table below:\n\n| $\\rho$ | MMLU $\\uparrow$ | WMDP-Bio $\\downarrow$ | WMDP-Cyber $\\downarrow$ | $\\Delta$Acc. $\\downarrow$ |\n| --- | --- | --- | --- | --- |\n| **5e-2** | 0.3366 (−0.2444) | 0.2710 (−0.3660) | 0.2441 (−0.1959) | -0.0755 |\n| **5e-3** | 0.5535 (−0.0275) | 0.2655 (−0.3715) | 0.2587 (−0.1813) | -0.5253 |\n| **5e-4** | 0.5601 (−0.0209) | 0.2727 (−0.3643) | 0.2506 (−0.1894) | -0.5328 |\n| **5e-5** | 0.5644 (−0.0166) | 0.2930 (−0.3440) | 0.2330 (−0.2070) | -0.5344 |\n\nThe table format is consistent with Table 4. As $\\rho$ increases, harmful information decreases; however, this also leads to a reduction in MMLU performance. We observe that the best performance is achieved at $\\rho =$ 5e-5.\nWe will definitely include an ablation section in the modified version to provide further insight into the contributions of different components of our method. Thank you again for your thoughtful comment.\n\n---\n\n**[W2] Deeper analysis of FF and IU**\n\nFirst of all, both FT/NG and FF/IU are weak unlearning approaches; however, they differ in their unlearning algorithms. Specifically, FT and NG are retraining-based methods, requiring multiple epochs and training losses. **In contrast, FF and IU do not rely on epochs**; instead, they use Hessian-based information and perturb the weights only once. Our proposed framework aligns with retraining-based methods. **Indeed, several works in the retraining-based line of research [4, S1] do not include FF/IU, as these are not retraining-based methods.** However, similar to recent work [6], we believe it is important to at least include FF/IU in our comparison table to provide ease of comparison for researchers.\nWe hope our method also contributes to the development of new unlearning frameworks based on Hessian-based algorithms such as FF/IU, which could relate to the insights presented in Lemma 2. We will incorporate this discussion into the Related Work section. Thank you for your feedback.\n\n---\n\n**[W3] Regarding the \"scalable and efficient\" claim**\n\nThank you for your comment. While it is true that our method requires two forward and backpropagation steps per iteration, compared to FT and NG, we would like to emphasize that FT and NG do not consistently achieve good unlearning performance, as demonstrated in Table 1. In this context, even though these methods may be computationally cheaper, they are not suitable for real-world applications due to their suboptimal unlearning performance. In contrast, our method demonstrates better unlearning performance.\n\nWe would like to point out that a similar issue was raised with sharpness-aware minimization [7, 22], which also requires two forward and backward passes. Despite initial concerns regarding its computational cost, SAM has become a popular technique for improving performance. Subsequent research has led to more efficient versions [S2, S3], which have reduced the computational requirements. Similarly, we expect that faster techniques will emerge based on the principles outlined in our work. Regarding the claim of \"scalability and efficiency,\" this specifically refers to the transition from Equation (10) to Equation (11). We will revise this section to ensure clarity in the modified version.\n\n---\n\n**[W4] Regarding the \"consistently outperforms existing methods\" claim**\n\nWe apologize for any confusion caused by the phrase \"consistently outperforms existing methods.\" We will revise this claim in the modified version to better align with the actual findings and avoid overstating our results.\n\n---\n\n**[W5] Validity of comparison methods**\n\nThank you for your comment. We would like to clarify that while FT and NG are earlier methods, they remain strong baselines that are still widely used in recent work [4, S1, S4]. **In fact, concurrent research [S4, S5, S6] presented at ICML 2025 uses FT, NG, and L1-sparse as main comparison methods.** L1-sparse was presented at NeurIPS 2023. We would also like to emphasize that our framework is flexible and can be integrated with more recent approaches [6, S5, S6, S7], as it follows a foundational training framework similar to FT and NG. **A notable example of this is SalUN, introduced at ICLR 2024, which we compare with our combined version in Table 2.** For large language model (LLM) unlearning, we have also considered state-of-the-art methods such as RMU, which was presented at ICML 2024. We hope this clarifies our choice of comparisons and strengthens the rationale for the experimental design. Thank you for your valuable feedback.\n\n---\n\n**[W6] Regarding the position of the proofs for Lemma 1 and Theorem 1**\n\nThank you for your valuable suggestion. Due to page limitations, we moved the proofs for Lemma 1 and Theorem 1 to the Appendix. However, we acknowledge the importance of these proofs in the derivation of UAM. In the modified version, we will move these proofs to the main text, as we have an additional content page available. Thank you for your comment.\n\n---\n**For minor issues,** we have corrected the typos in Line 129 (\"sorely\" to \"solely\") and Line 140 (\"an weight point\" to \"a weight point\"). We have also thoroughly reviewed the manuscript to address any other typographical errors. Thank you for your careful reading.\n \n---\n\n**Overall, we sincerely appreciate your positive evaluation of our paper, and we are grateful for your comments, which have significantly improved the readability of our work.**\n\n---\n**Supplementary References**\n\n- [S1] Huang, Zhehao, et al. \"Unified gradient-based machine unlearning with remain geometry enhancement.\" Advances in Neural Information Processing Systems 37 (2024): 26377-26414.\n- [S2] Liu, Yong, et al. \"Towards efficient and scalable sharpness-aware minimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- [S3] Mueller, Maximilian, et al. \"Normalization layers are all that sharpness-aware minimization needs.\" Advances in Neural Information Processing Systems 36 (2023): 69228-69252.\n- [S4] Gu, Hanlin, et al. \"Ferrari: federated feature unlearning via optimizing feature sensitivity.\" Advances in Neural Information Processing Systems 37 (2024): 24150-24180.\n- [S5] Ebrahimpour-Boroojeny, Ali, Hari Sundaram, and Varun Chandrasekaran. \"Not All Wrong is Bad: Using Adversarial Examples for Unlearning.\" Forty-second International Conference on Machine Learning.\n- [S6] Cai, Zikui, Yaoteng Tan, and M. Salman Asif. \"Targeted Unlearning with Single Layer Unlearning Gradient.\" Forty-second International Conference on Machine Learning.\n- [S7] Zhong, Zhengyi, et al. \"Unlearning through knowledge overwriting: Reversible federated unlearning via selective sparse adapter.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "aUY4b3pYC8", "reviewer": "Reviewer_NPr1", "summary": "This paper introduces a novel unlearning framework, Unlearning-Aware Minimization (UAM). UAM effectively removes forget set information while preserving retain set information by min-max optimization which involves inner maximization and outer minimization.", "questions": "- Although it showed superior unlearning performance compared to conventional methods, it seems not to work in a specific scenario where forget gradient is highly correlated with retain gradient. For example, in random data forgetting, if forget gradient from class 1 and retain gradient from class 1 may have same direction, then proposed objective function operates weirdly. Inferior unlearning performance in random data forgetting may be caused by this kind of reason. I want to ask the opinion of the authors about this concern.\n- Typos: In caption of Figure 1, gradient ascent (GA) is used instead of negative gradient (NG). In Table 1 and Table 5, some values in blue are not matched with the actual difference from Retrain. Some values even disturb interpreting overall results.", "limitations": "- UAM introduces additional hyperparameters, such as γ and ρ, in addition to the learning rate. It may need efforts to search optimal hyperparameters for each task, dataset, and model.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength:\n- This paper proposes a novel unlearning method, Unlearning-Aware Minimization, which effectively remove forget data while maintaining general model performances.\n- Proposed methods are theoretically well described.\n- Proposed methods can be easily extended to other domains, such as multiple-choice question-answering using large language models.\n\n### Weakness:\n- Proposed methods introduce additional hyperparameters, such as γ and ρ, which needs to be tuned.\n- In image classification, comparison is limited to the early unlearning methods. Comparison with recent advanced unlearning methods will strengthen the effectiveness of UAM.", "was_revised": true, "final_justification": "The rebuttal addresses most concerns, however, it remains unclear why UAM is less effective than some comparison methods, such as $l1$-sparse, in random-data forgetting.\n\nTherefore, the rating remains unchanged at borderline accept.\n\nIf the paper is accepted, it is recommended to revise the title \"Unlearning-Aware Minimization\", as it is unclear and does not accurately reflect the paper’s content.", "rebuttal": "We appreciate the reviewer’s insightful evaluation of the strengths and weaknesses of our paper. We believe that your feedback provides valuable perspectives that will help enhance the quality and clarity of our work.\n\nBelow, we provide a detailed response addressing each of the weaknesses [W#] and questions [Q#], with references to the main paper [#] as well as new supplementary references [S#].\n\n---\n\n**[W1] Sensitivity analysis on $\\rho$ and $\\gamma$**\n\nThank you for your valuable suggestion. First of all, we would like to note that an ablation study on Equation (8) with $\\rho$ and Equation (11) with $\\gamma$ is provided in Appendix B. However, we agree that including a sensitivity analysis on these hyperparameters under the main experimental settings would be valuable for the readers. The table below summarizes the results of the sensitivity experiment on $\\rho$ and $\\gamma$ for class-wise forgetting on CIFAR-10.\n\n| **$\\rho$** | **RA** | **FA** | **TA** | **$\\Delta$Acc. $\\downarrow$** | **MIA-Eff.** |\n| --- | --- | --- | --- | --- | --- |\n| **5e-3** | 99.99±0.00 (0.01) | 0.00±0.00 (0.00) | 94.32±0.66 (1.00) | 1.01 | 100.00±0.00 (0.00) |\n| **5e-2** | 100.0±0.00 (0.01) | 0.00±0.00 (0.00) | 94.51±0.63 (0.81) | 0.82 | 100.00±0.00 (0.00) |\n| **5e-1** | 100.0±0.00 (0.01) | 0.01±0.04 (0.01) | 93.33±1.08 (1.98) | 1.99 | 100.00±0.00 (0.00) |\n| **1e-1** | 99.65±0.31 (0.35) | 0.01±0.04 (0.01) | 92.66±1.12 (2.66) | 3.02 | 99.99±0.03 (0.01) |\n\n| **$\\gamma$** | **RA** | **FA** | **TA** | **$\\Delta$Acc. $\\downarrow$** | **MIA-Eff.** |\n| --- | --- | --- | --- | --- | --- |\n| **0** | 99.99±0.00 (0.00) | 63.59±6.13 (63.59) | 94.69±0.54 (0.63) | 64.22 | 95.80±2.66 (4.20) |\n| **1** | 99.99±0.01 (0.01) | 0.03±0.07 (0.03) | 94.57±0.63 (0.75) | 0.78 | 100.00±0.00 (0.00) |\n| **2** | 100.0±0.00 (0.01) | 0.00±0.00 (0.00) | 94.51±0.63 (0.81) | 0.82 | 100.00±0.00 (0.00) |\n\nThe table format is consistent with that of Table 1, with the only difference being the omission of the time column, as these results are based on the same computational setup. For $\\rho$, **our method demonstrates stable performance across a range of $\\rho$ values, with $\\Delta$Acc. significantly outperforming previous unlearning frameworks (e.g., FT: 43.63, NG: 24.45, FF: 10.22, IU: 13.32).** Regarding $\\gamma$, as noted in Appendix B, using a positive value of $\\gamma$ is crucial for achieving better $\\Delta$Acc. Specifically, $\\gamma=0$ implies that the Hessian information in Equation (8) is entirely ignored, which omits the core component of our method. **Using $\\gamma = 2$ and $\\rho = 0.05$ or $0.5$ results in stable performance across all domains and tasks.** The experimental results for LLM unlearning are presented in the table below:\n\n| $\\rho$ | MMLU $\\uparrow$ | WMDP-Bio $\\downarrow$ | WMDP-Cyber $\\downarrow$ | $\\Delta$Acc. $\\downarrow$ |\n| --- | --- | --- | --- | --- |\n| **5e-2** | 0.3366 (−0.2444) | 0.2710 (−0.3660) | 0.2441 (−0.1959) | -0.0755 |\n| **5e-3** | 0.5535 (−0.0275) | 0.2655 (−0.3715) | 0.2587 (−0.1813) | -0.5253 |\n| **5e-4** | 0.5601 (−0.0209) | 0.2727 (−0.3643) | 0.2506 (−0.1894) | -0.5328 |\n| **5e-5** | 0.5644 (−0.0166) | 0.2930 (−0.3440) | 0.2330 (−0.2070) | -0.5344 |\n\nThe table format is consistent with Table 4. As $\\rho$ increases, harmful information decreases; however, this also leads to a reduction in MMLU performance. We observe that the best performance is achieved at $\\rho =$ 5e-5. Due to the character limit, we are unable to include the ablation on $\\gamma$ on LLM unlearning.\nWe will definitely include an ablation section in the modified version to provide further insight into the contributions of different components of our method. Thank you again for your thoughtful comment.\n\n---\n\n**[W2] Validity of comparison methods**\n\nThank you for your comment. We would like to clarify that while FT and NG are earlier methods, they remain strong baselines that are still widely used in recent work [4, S1, S2]. **In fact, concurrent research [S2, S3, S4] presented at ICML 2025 uses FT, NG, and l1-sparse as main comparison methods.** L1-sparse was presented at NeurIPS 2023, and SalUN, which we compared in Table 2, was introduced at ICLR 2024. Moreover, we argue that our framework can be integrated with recent methods [6, S3, S4, S5], as it is a fundamental training framework similar to FT and NG. For large language model (LLM) unlearning, we have also considered state-of-the-art methods such as RMU, which was presented at ICML 2024.\n\n---\n\n**[Q1] Regarding the case when forget gradient and retain gradient are similar**\n\nThank you for your insightful suggestions. Inspired by your idea, we first investigate the cosine similarity between the forget gradient, $\\nabla L(w, D_f)$, and the retain gradient, $\\nabla L(w, D_r)$, under random-data forgetting. To do this, with a fixed random seed (seed=42), we observe that the average cosine similarity is higher under random-data forgetting than under class-wise forgetting. Specifically, the cosine similarity in random-data forgetting is greater than that in class-wise forgetting. **However, it is noteworthy that the cosine similarity does not approach 1.** Instead, it fluctuates between -0.25 and 0.50 (unfortunately, due to reviewer guidelines, we are unable to include the figure here).\n\nUnder the random-data forgetting setting, **we also observe that our method successfully disentangles the gradients.** The table below illustrates the range of cosine similarity across iterations during our unlearning:\n\n| Iteration | 0-500 | 500-1000 | 1000-1500 |\n| --- | --- | --- | --- |\n| Cosine value range | [-0.30,0.50] | [-0.25,0.25] | [0.00, 0.15] |\n\nTherefore, our method shows its effectiveness in identifying implicit features where the forget and retain gradients can be separated, leading to a decrease in the moving average of the cosine similarity. This suggests that our method is particularly useful when it is possible to distinguish between features to be forgotten and those to be retained.\n\nNow, let’s consider a thought experiment where $\\nabla L(w, D_r)$ and $\\nabla L(w, D_f)$ are identical for all iterations. This situation implies that the distributions of $D_r$ and $D_f$ are identical, making it impossible to disentangle these gradients. In this case, the forget accuracy should ideally not decrease, since any decrease would suggest different distributions. Interestingly, when the gradients are identical, the second term in (12) becomes constant. Only the loss $L(w, D_r)$ is primarily minimized. Therefore, even in this scenario, we argue that our method converges towards the optimal solution. This discussion can be verified by the results in Table 1, where our method shows stable performance across different random seeds, with a reduced accuracy gap. We appreciate your valuable comments and will include this discussion in the main text of the paper.\n\n---\n\n**[Q2] Typos and blue numbers in Tables**\n\nThank you for your comment. First, we would like to clarify that the blue values represent the average of the absolute differences between the \"Retrain\" method and each other method across different classes (or seeds). Therefore, the difference between the average values is not identical to the blue values. To explain this more clearly, let $v$ represent the value of each measure. Then, the following relationship holds: $\\mathbb{E}[|v_{retrain}-v_{method}|] \\neq \\mathbb{E}[v_{retrain}] - \\mathbb{E}[v_{method}].$ We understand that this could confuse the readers, and we will make sure to clarify this point in the final version of the paper.\n\nFor typos, including GA, we have thoroughly reviewed the manuscript to address any other typographical errors. Thank you for your careful reading.\n\n---\n\n**Overall, we truly appreciate your positive evaluation of our paper, and we are grateful for your detailed comments, which have significantly improved the novelty of our work.**\n\n---\n\n**Supplementary References**\n\n- [S1] Huang, Zhehao, et al. \"Unified gradient-based machine unlearning with remain geometry enhancement.\" Advances in Neural Information Processing Systems 37 (2024): 26377-26414.\n- [S2] Gu, Hanlin, et al. \"Ferrari: federated feature unlearning via optimizing feature sensitivity.\" Advances in Neural Information Processing Systems 37 (2024): 24150-24180.\n- [S3] Ebrahimpour-Boroojeny, Ali, Hari Sundaram, and Varun Chandrasekaran. \"Not All Wrong is Bad: Using Adversarial Examples for Unlearning.\" Forty-second International Conference on Machine Learning.\n- [S4] Cai, Zikui, Yaoteng Tan, and M. Salman Asif. \"Targeted Unlearning with Single Layer Unlearning Gradient.\" Forty-second International Conference on Machine Learning.\n- [S5] Zhong, Zhengyi, et al. \"Unlearning through knowledge overwriting: Reversible federated unlearning via selective sparse adapter.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "Ji2YbpnJfR", "reviewer": "Reviewer_mZMi", "summary": "This paper formulates machine unlearning as a minimax optimization problem. Specifically, it first maximizes the “forget” loss over a local delta-neighborhood of the parameters \nw, and then minimizes the “retain” loss based on that adversarial perturbation. Drawing inspiration from Sharpness-Aware Minimization, the inner maximization admits a closed-form approximation (Theorem 1), and the authors further leverage a first-order Taylor expansion for computational efficiency.", "questions": "No more questions.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe manuscript is well structured and clearly written, with a thorough and precise overview of related work.\n\nWeaknesses\n\n1. It omits discussion of a closely related adversarial training approach to unlearning [1]. Reference [1] also considers a minimax optimization formulation to address unlearning problems. Could the authors provide further explanations on how their approach differs from [1]?\n\n2. The regularizer in Equation (2) compares losses at the optimal and approximate solutions but does not directly penalize the distance between them — why is that? Furthermore, the criterion for choosing $\\epsilon$ to ensure that $w^*$ lies within the $\\delta$-ball is unclear.\n\n3. The experimental comparisons do not include several recent baselines [2–5], particularly in Tables 1 and 3. Moreover, there should be an ablation study on the perturbation range $\\rho$.\n\n4. The procedure for generating the optimization trajectories shown in Figure 1 is not clear. Which model and data were used to obtain these trajectories?\n\nReferences\n\n[1] Discriminative Adversarial Unlearning\n\n[2] Large Language Model Unlearning via Embedding-Corrupted Prompts\n\n[3] MUSE: Machine Unlearning Six-Way Evaluation for Language Models\n\n[4] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning\n\n[5] Who's Harry Potter? Approximate Unlearning in LLMs", "was_revised": true, "final_justification": "The authors have addressed all my concerns in their rebuttal and during the communication period. For this reason, I have raised the score from 2 to 4.", "rebuttal": "We appreciate the reviewer’s evaluation of the strengths and weaknesses of our paper. Below, we provide a detailed response addressing each of the weaknesses [W#], with references to the main paper [#] as well as new supplementary references [S#].\n\n---\n\n**[W1] Comparison to [S1]**\n\nThank you for suggesting the reference [S1]. However, we believe the work in [S1] differs significantly from our approach. In [S1], adversarial unlearning is framed in the context of Generative Adversarial Networks (GANs), where two distinct networks (attacker and defender networks) are used. This differs fundamentally from our approach, which involves a min-max optimization procedure applied to a single network’s weight parameters, similar to sharpness-aware minimization [7, 22]. Moreover, adversarial robustness focuses on maximizing the input perturbation and minimizing an outer loss function, such as cross-entropy. In contrast, our work emphasizes optimization directly on the model parameters, which sets it apart from the traditional adversarial robustness framework.\n\n---\n\n**[W2] Regarding penalizing distance to $w^*$ and clarity of algorithm**\n\nEquation (2) represents our theoretical analysis for integrating prior unlearning methods. In machine unlearning, the optimal solution **$w^*$** is assumed to be unknown due to the significant cost of obtaining it. Therefore, prior works [6, 22] focus on updating the model to remove the influence of forget data. In our work, rather than ignoring it, we replace the optimal solution with a surrogate weight based on the characteristic of **$w^*$**. Specifically, in equations (4) and (5), $\\epsilon$ represents the concept that the surrogate weight should incur a higher forget loss than the current weight. Within the $\\rho$-neighborhood in the weight space, there exists a weight with a higher loss, as has been previously discussed in other works [7, 22]. As evidence of the effectiveness of our approach, our method achieves a higher forget loss, as illustrated in Figures 1 and 2.\n\nFor the clarity of our algorithm, we would like to emphasize that other reviewers, such as Reviewer MRvE, who stated \"This paper presents a novel machine unlearning framework called Unlearning-Aware Minimization (UAM), which formulates unlearning as a min-max optimization problem,\" Reviewer QDnV, who mentioned \"The key innovation of this paper lies in formulating the machine unlearning problem within a min-max optimization framework, which leads to promising empirical performance,\" and Reviewer NPr1, who remarked \"Proposed methods are theoretically well described,\" all agreed with the contributions of our paper.\n\n---\n\n**[W3] Regarding other baselines and ablation study on $\\rho$**\n\nThank you for suggesting the references [S2, S3, S4, S5]. While we appreciate these recommendations, **we believe that these works are not directly comparable** to ours for the following reasons:\n\n- **[S2]**: This work does not involve updating the model's parameters directly, which makes a direct comparison to our method unfair. Our approach specifically updates the weight distribution of the model, which is a key distinction.\n- **[S3]**: This is a benchmark dataset, not a baseline model, making it inappropriate for a direct comparison. We chose to use WMDP datasets in our work because they are widely used and facilitate easier comparison with other works in the field.\n- **[S4] and [S5]**: These approaches do not use a \"retain set.\" In [S4], the focus is solely on the \"forget set\" during unlearning, whereas our method utilizes a general corpus as the \"retain set.\" Similarly, [S5] proposes an unlearned model through interpolation between the target model and a reinforced model, but it does not involve a retain set. Therefore, a direct comparison with our approach is not feasible.\n\nGiven these distinctions, we believe that the methods mentioned above are not as relevant to our work as RMU and others in Table 4, which align more closely with the goals and methods we propose. We would like to emphasize that **our goal is to present a unified framework that can be applied across various domains**, a feature that we consider a strength of our approach as **Reviewer MRvE noted, \"UAM does not rely on specific loss functions or architectures, allowing it to be applied broadly across domains including image classification and LLM unlearning.\"** We will include a discussion of these works in the revised version of the paper.\n\n**For the ablation study,** we would like to note that an ablation study on Equation (8) with $\\rho$ is already provided in Appendix B. However, we agree that including a sensitivity analysis on these hyperparameters under the main experimental settings would be valuable for the readers. The table below summarizes the results of class-wise forgetting on CIFAR-10.\n\n| **$\\rho$** | **RA** | **FA** | **TA** | **$\\Delta$Acc. $\\downarrow$** | **MIA-Eff.** |\n| --- | --- | --- | --- | --- | --- |\n| **5e-3** | 99.99±0.00 (0.01) | 0.00±0.00 (0.00) | 94.32±0.66 (1.00) | 1.01 | 100.00±0.00 (0.00) |\n| **5e-2** | 100.0±0.00 (0.01) | 0.00±0.00 (0.00) | 94.51±0.63 (0.81) | 0.82 | 100.00±0.00 (0.00) |\n| **5e-1** | 100.0±0.00 (0.01) | 0.01±0.04 (0.01) | 93.33±1.08 (1.98) | 1.99 | 100.00±0.00 (0.00) |\n| **1e-1** | 99.65±0.31 (0.35) | 0.01±0.04 (0.01) | 92.66±1.12 (2.66) | 3.02 | 99.99±0.03 (0.01) |\n\nThe experimental results for LLM unlearning are presented in the table below:\n\n| $\\rho$ | MMLU $\\uparrow$ | WMDP-Bio $\\downarrow$ | WMDP-Cyber $\\downarrow$ | $\\Delta$Acc. $\\downarrow$ |\n| --- | --- | --- | --- | --- |\n| **5e-2** | 0.3366(−0.2444) | 0.2710(−0.3660) | 0.2441(−0.1959) | -0.0755 |\n| **5e-3** | 0.5535(−0.0275) | 0.2655(−0.3715) | 0.2587(−0.1813) | -0.5253 |\n| **5e-4** | 0.5601(−0.0209) | 0.2727(−0.3643) | 0.2506(−0.1894) | -0.5328 |\n| **5e-5** | 0.5644(−0.0166) | 0.2930(−0.3440) | 0.2330(−0.2070) | -0.5344 |\n\nThe table format is consistent with Table 4. As $\\rho$ increases, harmful information decreases; however, this also leads to a reduction in MMLU performance. We observe that the best performance is achieved at $\\rho = $ 5e-5. **We will definitely include an ablation section in the modified version to provide further insight into the contributions of different components of our method. Thank you again for your comment.**\n\n---\n\n**[W4] Regarding Figure 1**\n\nIn lines 189-191, we provide a detailed explanation of Figure 1. Specifically, we design a simple synthetic 2D optimization landscape with $w=[w_1,w_2]$, where the forget and retain losses are derived from rotated quadratic forms. The optimization is initialized at the point that minimizes the sum of the forget and retain losses, representing a pre-trained model. For further details, we refer the reader to Appendix C.1.\n\n---\n**Overall, we sincerely appreciate your evaluation of our paper, and we are grateful for your comments. Thank you.**\n\n---\n\n**Supplementary References**\n\n- [S1] Sharma, Rohan, et al. \"Discriminative adversarial unlearning.\" arXiv preprint arXiv:2402.06864 (2024).\n- [S2] Liu, Chris, et al. \"Large language model unlearning via embedding-corrupted prompts.\" *Advances in Neural Information Processing Systems* 37 (2024): 118198-118266.\n- [S3] Shi, Weijia, et al. \"MUSE: Machine Unlearning Six-Way Evaluation for Language Models.\" *The Thirteenth International Conference on Learning Representations*.\n- [S4] Zhang, Ruiqi, et al. \"Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning.\" *First Conference on Language Modeling*.\n- [S5] Eldan, R., and M. Russinovich. \"Who’s Harry Potter? Approximate unlearning in LLMs, arXiv.\" *arXiv preprint arXiv:2310.02238* (2023)."}, {"review_id": "19YzUKsM1I", "reviewer": "Reviewer_QDnv", "summary": "In this paper, authors focus on the machine unlearning problem, which aims to eliminate the influence of forget data while preserving the performance on the retain data. The article points out that solely using negative gradient or fine-tuning methods fail to effectively balance the objectives of maximizing forget loss and minimizing retain loss. And authors propose a novel min-max optimization framework to consistently achieve higher forget loss and lower retain loss. Based on first-order Taylor expansion, the paper proposes an efficient solution to the above optimization and theoretically analyzes the validity of the update strategy.", "questions": "- In the related work 2.1, the authors mention that “our method is a min-max optimization framework rather than relying on dual-objective formulations.” What are the advantages of min-max optimization framework compared to dual-objective formulations?\n- From eq.3, it can be found that the variable in the minimization problem is w, which means the w* is fixed. And Eq.5 proposes a maximiaztion problem to estimate w*. Therefore, when we solve the maximization problem, the w* should be fix to w0+\\rho (\\nabla \\nabla L_f(w))/(||\\nabla \\nabla L_f(w)||_2^2). However, in the update process of eq.7, the w* is still treated as a variable. It seems the problem in eq.3 is not equal to that in eq.7.\n- Under random data setting, since the gradients calculated on retain data are similar to those calculated on forget data, it can be found that the results of UAM are relatively weaker, is there any solution to mitigate this problem?\n- In practical large-scale settings (e.g., LLMs), users may only provide forget sets. Is it realistic or necessary to assume full access to both retain and forget data partitions?\n- When using a first-order Taylor approximation, please specify the smoothness requirements for the function.\n- The graphical explanation in Figure 1 contains a typo: “GA”.\n- There is a grammatical error in line 113 on page 3: “where and Ω is a finite upper bound on the distance.”\n- The following sentence in the abstract may be ambiguous and could benefit from further clarification:“In this paper, we propose Unlearning-Aware Minimization (UAM), a novel min-max optimization framework that identifies weights with high loss on the forget data and uses their gradients to minimize the loss on the retain data.”", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n- The key innovation of this paper lies in formulating the machine unlearning problem within a min-max optimization framework, which leads to promising empirical performance. \n- By leveraging a first-order Taylor approximation, the authors transform a challenging bi-level optimization into a tractable update rule, enabling efficient gradient-based training. \n- The paper provides an intuitive geometric interpretation, illustrating how the alignment between the gradients of forget and retain data affects the optimization process, thereby enhancing the clarity and interpretability of the proposed method.\n\nWeakness:\n\nWhile the paper presents a novel and empirically effective framework for machine unlearning, several important aspects deserve further clarification and theoretical grounding:\n- About assumption and setting: The assumption that both the forget and retain data are known and accessible may not hold in realistic large-scale settings such as LLMs, where only the forget set is identifiable. This limits the method’s applicability in more practical scenarios.\n\n- About the technique: The method relies on an approximate optimization using a surrogate objective. However, it remains unclear whether the solution trajectory truly converges to the intended optimal parameter w^*,and how large the approximation gap is between the proxy and the true formulation. The clarification of the real gap is necessary for readers to understand the method.\n\n- About the theory for the effectiveness of model unlearning.\nThe effectiveness of forgetting (i.e., removing the influence of the forget set) and maintaining performance on the retain set is empirically verified, but lacks formal guarantees. Does the proposed objective ensure meaningful unlearning under general conditions? Is there a generalization bound or any guarantee of worst-case behavior?", "was_revised": true, "final_justification": "The response has addressed my concerns. I tend to increase my score to 4.", "rebuttal": "We appreciate the reviewer’s insightful evaluation of the strengths and weaknesses of our paper. We believe that your feedback provides valuable perspectives that will help enhance the quality and clarity of our work.\n\nBelow, we provide a detailed response addressing each of the weaknesses [W#] and questions [Q#], with references to the main paper [#] as well as new supplementary references [S#].\n\n---\n\n**[W1, Q4] Regarding the setting using both forget and retain data**\n\nWe investigated all recent papers on LLM and Vision unlearning over the past two years from top-tier conferences (NeurIPS, ICML, ACL, EMNLP, NAACL, CVPR, ICCV). **In total, we reviewed 71 papers, a detailed list of which will be provided as a comment.** Among these, 39 papers focus primarily on approximate unlearning (excluding certified unlearning), which is also the main theme of our work.\n\n| **Approximate unlearning** | **Use Retain & Forget** | **Forget Only** | **Retain Only** | **Others** |\n| --- | --- | --- | --- | --- |\n| **#Papers** | 27 | 3 | 5 | 4 |\n\nAmong these, 29 papers utilize both the retain and forget datasets. While access to both datasets may be limited in practical scenarios, we argue that it is a common assumption in the current research field to use both.\n\nNonetheless, we agree that evaluating our method in a more restricted setting is valuable for broader applicability. Below, we conducted experiments with a reduced number of forget data for class-wise forgetting on CIFAR-10. Random data forgetting is more similar to instance-wise forgetting, which does not align with the limited forget data. We used only 10% of each removed class and compared methods that are influenced by adjustments to the forget dataset, namely NG and IU.\n\n| 10% | **RA** | **FA** | **TA** | **MIA-Eff.** |\n| --- | --- | --- | --- | --- |\n| NG | 94.05±3.28 | 21.09±1.41 | 87.96±3.31 | 1.10±20.70 |\n| IU | 99.93±0.15 | 81.20±14.54 | 95.01±0.83 | 19.55±55.21 |\n| UAM | 99.99±0.01 | 0.00±0.00 | 94.75±0.66 | 100.00±0.00 |\n\nInterestingly, despite reducing the size of the forget dataset, **our method still demonstrates high performance, whereas the performance of NG and IU degraded significantly.** In particular, IU shows high forget accuracy due to the smaller forget dataset, which leads to an inaccurate approximation of the weight adjustment direction. **We believe that this experiment effectively highlights the robustness of our method.** We sincerely appreciate your insightful comments and will ensure that the revised version addresses this phenomenon.\n\n---\n\n**[W2, Q2] Regarding the optimization formula**\n\nAccording to [S7], approximate unlearning can be classified into two categories: strong unlearning and weak unlearning. Strong unlearning seeks to match the model parameters or their distribution exactly, while weak unlearning aims to produce outputs that are similar to those of a retrained model. Our work aligns with weak unlearning, rather than strong unlearning. As noted in other weak unlearning methods [6, 16, 20], accurately quantifying the gap between the original and retrained models is inherently difficult within this framework. For a more detailed discussion of the theoretical guarantees, we refer the reviewer to [W3]. In this context, the optimal parameter $w^*$ cannot be directly inferred under the weak unlearning framework. As a result, **we utilize a surrogate parameter, $\\hat{w}$.** It is important to clarify that $\\hat{w}$ is not a fixed parameter; rather, **it can be any point within the set $ \\hat{w} \\in B^{\\epsilon}_{\\Omega} (w_0; D_f) $**. **We argue that this surrogate parameter provides valuable information that can guide the model towards the optimal point $w^*$.** This continuous optimization process is closely related to concepts such as generalization [7, 22] and adversarial robustness [10, 29, 37]. We will revise this section to make these points clearer. Once again, thank you for your comment.\n\n---\n\n**[W3] Lacks formal guarantees on generalization bound or any guarantee of worst-case behavior.**\n\nOur work focuses on approximate unlearning with weak unlearning [6, 16, 23], which addresses unlearning by empirically retraining models. **Theoretical guarantees, such as generalization bounds, are more commonly associated with certified unlearning [S1, S5, S6],** a distinct area that combines approximate unlearning with differential privacy and remains less explored in machine learning [S1].\n\nWhile approximate unlearning methods lack formal guarantees, they have gained significant attention (e.g., L1-sparse [16] with 156 citations, SalUN [6] with 189 citations, and RMU [23] with 219 citations). Some works have attempted theoretical guarantees for approximate unlearning algorithms ([S3], [S4]), but these usually offer only upper bounds on information retained in model weights, rather than strict guarantees. **This highlights the challenge in providing formal guarantees for non-certified unlearning.** As noted by [20], unlearning lacks a well-defined formal definition, complicating the development of standardized evaluation metrics. Furthermore, most theoretical work has focused on certified unlearning, emphasizing that formal guarantees for approximate unlearning remain an open problem.\n\nHowever, [S2] suggests that certified unlearning methods can be adapted to approximate unlearning, particularly for convex models, which could improve their applicability to deep neural networks. This points to a potential integration of both approaches. **We believe that our investigation into the Hessian contributes to advancing certified unlearning**. Thank you for your feedback.\n\n---\n\n**[Q1] Regarding the statement in Section 2.1:**\n\nIn the sentence, we would like to clarify that we have developed a novel unlearning algorithm based on min-max optimization. The statement was not meant to argue that the min-max optimization approach is superior to dual-objective formulations. We will revise this part of the manuscript to make this distinction clearer. Thank you for your detailed comment.\n\n---\n\n**[Q3] Regarding similarity between forget and retain gradients**\n\nThank you for your insightful suggestions. Inspired by your idea, we first investigate the cosine similarity between the forget gradient, $\\nabla L(w, D_f)$, and the retain gradient, $\\nabla L(w, D_r)$, under random-data forgetting. To do this, with a fixed random seed (seed=42), we observe that the average cosine similarity is higher under random-data forgetting than under class-wise forgetting. Specifically, the cosine similarity in random-data forgetting is greater than that in class-wise forgetting. **However, it is noteworthy that the cosine similarity does not approach 1.** Instead, it fluctuates between -0.25 and 0.50.\n\nUnder the random-data forgetting setting, **we also observe that our method successfully disentangles the gradients.** The table below illustrates the range of cosine similarity across iterations during our unlearning:\n\n| Iteration | 0-500 | 500-1000 | 1000-1500 |\n| --- | --- | --- | --- |\n| Cosine value range | [-0.30,0.50] | [-0.25,0.25] | [0.00, 0.15] |\n\nTherefore, our method shows its effectiveness in identifying implicit features where the forget and retain gradients can be separated. This suggests that our method is particularly useful when it is possible to distinguish between features to be forgotten and those to be retained.\n\nBeyond this result, the poor performance on random-data forgetting is a common phenomenon in machine unlearning [6, 16], as it is more challenging due to the high similarity between data distributions. In our framework, using a higher value of $\\rho$ in random-data forgetting leads to more robust performance, as noted in Appendix C.2. Thank you for your detailed comment.\n\n---\n\n**[Q5] Regarding the first-order Taylor approximation**\n\nThank you for your comment. **We would like to emphasize that the linearity assumption is a technique commonly employed in generalization [7, 22] and adversarial robustness [10, 29, 37].** While this assumption may not hold in more complex models and datasets, these methods [7, 10, 22, 29, 37] have shown effective optimization in their respective domains. Specifically, sharpness-aware minimization [7, 22] demonstrates high performance when $\\rho$ is in the range of $\\rho \\in$ {$0.005, 0.05, 0.5$}. Similarly, our method demonstrates stable performance across the range of $\\rho$ values. The table below summarizes the results of class-wise forgetting on CIFAR-10.\n\n| **$\\rho$** | **RA** | **FA** | **TA** | **$\\Delta$Acc. $\\downarrow$** | **MIA-Eff.** |\n| --- | --- | --- | --- | --- | --- |\n| **5e-3** | 99.99±0.00 (0.01) | 0.00±0.00 (0.00) | 94.32±0.66 (1.00) | 1.01 | 100.00±0.00 (0.00) |\n| **5e-2** | **100.0±0.00 (0.01)** | **0.00±0.00 (0.00)** | **94.51±0.63 (0.81)** | **0.82** | **100.00±0.00 (0.00)** |\n| **5e-1** | 100.0±0.00 (0.01) | 0.01±0.04 (0.01) | 93.33±1.08 (1.98) | 1.99 | 100.00±0.00 (0.00) |\n| **1e-1** | 99.65±0.31 (0.35) | 0.01±0.04 (0.01) | 92.66±1.12 (2.66) | 3.02 | 99.99±0.03 (0.01) |\n\nThe table format is consistent with Table 1, except for the omission of time, as the results are based on the same computational setup. **Our method shows stable performance for a range of $\\rho$, with $\\Delta$Acc. values significantly better than those of previous unlearning frameworks (e.g., FT: 43.63, NG: 24.45, FF: 10.22, IU: 13.32).** We will definitely include an ablation section, including more results on various settings, in the modified version. Thank you again for your thoughtful comment.\n\n---\n\n**[Q6-8] Typos and ambiguous sentences**\n\nWe have corrected all typos and thoroughly reviewed the manuscript to address any other errors. Thank you for your careful reading.\n\n---\n\n**Overall, we sincerely appreciate your detailed evaluation of our paper, and we are grateful for your comments, which have significantly improved the novelty of our work.**"}]

Author rebuttal:

Meta-review: **Summary:** This paper proposes Unlearning-Aware Minimization (UAM), a min-max optimization framework for machine unlearning that identifies weight perturbations maximizing forget loss, and then minimizes retain loss based on these surrogate weights. The method is inspired by sharpness-aware minimization and provides both a theoretical justification and efficient optimization. Experiments across image classification datasets and LLM-based unlearning tasks demonstrate that UAM outperforms strong baselines, achieving effective forgetting while preserving utility.

**Decision:** During the review and rebuttal process, several concerns were raised, including the reliance on first-order approximations, limited comparisons to the most recent baselines, scalability and efficiency claims, and weaker performance under random-data forgetting. The authors actively engaged with reviewers, adding ablations on hyperparameters, integrating newer baselines (e.g., SEMU, AMUN), and providing further analyses of gradient similarity and optimization behavior. Reviewers generally acknowledged these efforts, with some explicitly raising their scores after rebuttal. While certain limitations remain, the overall consensus is that the proposed framework makes a solid and timely contribution to the growing area of machine unlearning. I therefore recommend acceptance.
Decision: Accept (poster)