Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Authors: Shiji Zhou
Tianbai Yu
Zhi Zhang
Heng Chang
Xiao Zhou
Dong Wu
Han Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Machine Unlearning; Multi-objective optimization
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We cast unlearning as constrained optimization (minimize unlearning subject to bounded utility loss) and propose implicit gradient-surgery that recovers the constrained solution with one backprop, enabling efficient, utility-preserving unlearning.

Abstract: Machine unlearning (MU) aims to efficiently remove sensitive or harmful memory from a pre-trained model. The key challenge is to balance the potential tradeoff between unlearning efficacy and utility preservation, which involves forgetting undesirable information as defined while maintaining the model's original performance. One potential way to tackle this problem is to use multi-objective optimization to jointly optimize both the unlearning and utility preservation objectives. However, existing multi-objective methods only guarantee finding a Pareto-optimal solution without fine-grained control, which causes under-optimization of the unlearning objective. To this end, we first model MU as a constrained optimization problem, that is, optimizing the unlearning objective under the constraint of a bounded increase for utility loss. We then show that solving this optimization problem is equivalent to unilateral gradient surgery on the unlearning objective. To resolve the additional computational cost brought by gradient surgery, we propose an implicit gradient surgery method, which approximates the solution to the aforementioned constrained optimization problem via only one backpropagation, thereby achieving efficient utility-preserving MU. Theoretically, we provide a tight convergence analysis of the algorithm. Empirically, our extensive experiments show that the proposed algorithm achieves better tradeoff results than existing baselines. Codes are available at https://github.com/anseryuer/EUPMU-Efficient-Utility-Preserving-Machine-Unlearning.

Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Shiji Zhou!:2y 12 Yu', Zhi Zhang', Heng Chang', Xiao Zhou', Dong Wu Han Zhao
"Intitute of Artificial Intelligence, Beihang University 2 Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University University of Illinois at Urbana-Champaign *University of Amsterdam Tsinghua University Yan'ron Technology Co.Ltd
Abstract
Machine unlearning (MU) aims to efficiently remove sensitive or harmful memory from pre-trained model. The key challenge is to balance the potential tradeoff between unlearning efficacy and utility preservation, which involves forgetting undesirable information as defined while maintaining the model's original perfor- mance. One potential way to tackle this problem is to use multi-objective opti- mization to jointly optimize both the unlearning and utility preservation objectives. However, existing multi-objective methods only guarantee finding a Pareto-optimal solution without fine- -raiina control, which causes under-optimization of the un- learning objective. To this end, we first model MU as a constrained optimization problem, that is, optimizing the unlearning objective under the constraint of bounded increase for utility loss. We then show that solving this optimization problem is equivalent unilateral gradient surgery on the unlearning objective. To resolve the additional computational cost brought by gradient surgery. we pro- pose an implicit gradient surgery method, which approximates the solution the aforementioned constrained optimization problem via only one backpropagation, thereby achieving efficient peiiity-peeeeiinggg MU. Theoretically, we provide a tight convergence analysis of the algorithm. Empirically, our extensive exper- iments show that the proposed algorithm achieves better tradeoff results than existing baselines. Codes are available at https: thub. com/ wwt.//oo//aaaauaaae anseryuer/ -ereeerriity-.eeeeiiiingg. wwwpe--aaaaaeeeeeee-iiiiiaaa Unlearning
Introduction
The growing capacity of large generative models 39 74] has inevitably led increasing concerns about their potential security risks. In particular, massive pre-training data from large models may contain privacy, copyright, and illegal information about individual users, which can inadvertently memorized through model parameters through training, posing risk of content leakage under model inversion attacks 21 71. Moreover, the high training costs of large models make addressing these issues in pre-trained models particularly challenging [52], since naive retraining is computationally infeasible. Consequently, both industry and academia are actively seeking efficient methods to enable the erasing of sensitive information at small cost.
Machine unlearning (MU) which aims to enable models efficiently remove memory of sensitive data, is a potential approach to meet the above goals. The current landscape of MU
Part of the work done while Shiji Zhou at Tsinghua University 1 "ooresponning to Shiji Zhou: zhoushiji25 @ buaa.edu.cn Shiji Zhou and Tianbai Yu have equal contributions
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.0
Parats PART
Pareto
Pareto Frent
Objective
0.8
Uslity Objective
6.8
uouote Uasee E
Unility Objjetivee
supito
0.4
0.2
02
g
2.6
0.2
0.8
0.4
Unleariing Objective
Unleaminn Objective
Unlearning Objective
(a) Linear
(b) MGDA
(C) BUPMU
Figure 1: A showcase where both linear scalarization and MGDA fail find a good tradeoff solution that balance the utility and unlearning objectives: (a) Linearization often leads to extreme solutions with deterioration on one objective: (b) The pre-trained model fully optimizes the utility objective, leaving little room to improve unlearning objective for MGDA algorithm that fairly optimizes all objectives; (c) Our EUPMU algorithm provides a tolerance for small degradation for utility objective, and returns better tradeoff solution.
research encompasses range of generative models, such as large language models (LLM)
image synthesis models [50]. and multi-modal generative frameworks [59]. These investigations
have showcased the capacity of MU eliminate specific data, including copyrighted patterns (22). fake personas [17]. and confidential information [60]. However, the unlitt-unl tradeoff is key issue in MU [47). where there is a fundamental tradeoff between enhancing the unlearning effect and maintaining the model's 's original performance. This often leads to a degradation of the model's performance when unlearning sensitive information. Current solutions involve incorporating retaining loss, calculated from the retained portion of the training data, into the unlearning training process, which can alleviate some of the issues with utility degradation [ [8]. However, simply combining unlearning and retaining objectives that have inherent conflict fails to find a balanced solution, as it has been proved multitask learning literature
as demonstrated in Figure [(a)
A potential way to mitigate the conflicts between the two targets is to employ a multi-objective optimization (MOO) (77). However, existing MOO methods cannot control the converged solutions in a fine-grained way. Specifically, pre-trained models have already been thoroughly optimized for utility, hence it is already near Pareto optimal solution on the Pareto front between unlearning and utility. Since there is little room for further optimization of the utility objectives, directly applying MOO methods that often fairly optimize all objectives to the unlearning problem may result in insufficient optimization of the unlearning objectives, as demonstrated in Figure [[b) Moreover, most existing MOO methods require derivatives for each objective, doubling the number of backpropagation iterations compared to linear weighting methods, thereby doubling the computational cost. This contradicts the high efficiency and low-cost demand of unlearning.
To tackle the challenges, this paper makes the following principal contributions:
1) To better address the utility- cullitt-unli tradeoff, first formulate the Urility-rreeerving Unlearn- ing Problem (UPUP), where the preservation of utility is formulated as a constraint on the increase in retaining loss at each step, with the goal of maximizing the decrease in unlearning loss under this constraint, thereby optimizing the unlearning objective while preserving utility. Solving UPUP leads to gradient method equivalent to Unilateral Gradient Surgery, which subtracts the components of the unlearning gradient that conflict with the retaining gradient. This ensures that utility degradation is within controllable range while maxing the erasing of target information.
2) Explicit gradient surgery inevitably doubles the gradient computation. This paper first presents an Efficient Stility-Preservii Machine Unlearning (EUPMU) method with Efficient Implicit Gradient Surgery, utilizing the essence of gradient surgery being equivalent to dynamic linear weighting. Solv- ing the weights through first-order approximation, requires only one pass of gradient computation, thus saving up to 50% of the main computational costs. Furthermore, we provide a multi-objective convergence analysis for the proposed algorithm, proving that the algorithm can efficiently converge to Pareto optimal (or stationary) solution, demonstrating that the algorithm can achieve sufficient unlearning while preserving utility.
3) We conducted comprehensive experiments on tasks including image classification and image generation. Both numerical and visual results demonstrated significant improvements, effectively

---

proving that our method can fully optimize the unlearning objective while maintaining utility to the greatest extent. This further validates the design and theoretical outcomes of our approach.
Background
We begin by introducing the basic concepts of machine unlearning, elucidating the utility-unlearning challenge encountered by existing methods. We then present the concept of Multi-Objective Optimiza- tion (MOO) as a potential solution this problem and discuss why MOO cannot be directly applied to address the challenges in MU. We defer detailed comparison with related works in Appendix A
2.1 Machine Unlearning
Given the training dataset DD = (z, (2;; 31) )))) the original/pree-raained model with parameters On is encapsulated by the following optimization:
0o = arg min minE,-p((;;0))
MU focuses on eliminating the influence of specific data subset Dy C D, which is the forget- ting dataser that may include harmful or sensitive information. The goal is efficiently derive an unlearned model 02 by finetuning the original parameters do. while maintaining the model's performance on the retaining dataset D,
Dz.
To counteract the impact of the forgetting dataset D,, MU algorithms often establish objectives that aim to reverse the effects of the initial training These include objectives such as inverse loss [61] and random labeling [24]. We denote this corrective objective the unlearning objective (. Furthermore, to maintain performance on the remaining data, most MU methods incorporate an objective that mimics retraining samples from the retaining dataset D,.. This is known as the retaining objective & (0). Typically, these objectives are linearly combined into one objective (18]
(0) + (O),
where À is hyperparameter balancing the two parts.
Utility-Unlearning Challenge. While linearization is straightforward to implement, may lead to performance deterioration or insufficient unlearning, due to the inherent conflict between the unlearning and retaining objectives [72 (5). The underlying theoretical reason may be that fixed linear weights fails to find solution that balances the objectives [34] shown in Figure I (a).
2.2 Multi-Objective Optimization
Multiple-ohjective optimization (MOO) aims to optimize multiple objectives simultaneously
min = = ()) e T.))
where m 2 denotes the number of objectives, and C R + R is the i-th loss function. Denote A, to be the (m dimeenssoaaa probability simplex. The concept of Pareto optimalittyssatiinnay I2 is introduced to determine whether a solution to MOO is optimal/critical
Definition 2.1. Pareto Optimality: For any two solutions 0, 0', we say that dominates O', denoted as 0 < 0' or 0' > 0, if 8 (0) < (00') for all i, and there exists one i such that (0) < (00). A
solution 0* E called Pareto optimal if is not dominated by any other solution. Pareto Stationary: A solution 0 is called Pareto stationary if there exists À Am such that V. (o
0.
Typical MOO methods like MGDA 14, PCGrad 699 and other variants 46 76
45
28
aim to [m] Using
search for direction d that is not conflicting with each gradient, i.e., V d > i
such a non-confiicting direction dę as the update direction is shown to get better tradeoff performance.
Can MOO Solve the optimization process and may offer a solution to the utility-unlearning challenge in MU. However,
Tradcoff? MOO addresses conflicts among objectives during
'We here consider approximate unlearning, since exact unlearning often needs to retrain the model, which is impractical for large models due to the high retraining cost.

---

the optimization goals of MOO and MU are not aligned. making existing MOO methods inadequate for unlearning in pre-trained models. Specifically, pre-trained models have been thoroughly optimized for the utility objective, while the unlearning objective has not been optimized. Therefore, the goal of unlearning is to fully optimize the unlearning objective while maintaining utility to the greatest extent possible, albeit with some minor degradation if necessary, like the solution in Figure 1 (c). However, the primary goal of typical MOO methods to identify an optimization path that benefits all objectives simultaneously. Given that there is little room for further improvement in utility objectives, directly applying MOO methods to optimize utility and unlearning objectives fairly could lead to inadequate optimization of the unlearning objective, as shown in Figure [] (b). In addition, MOO often requires gradient computation for each objective, which doubles the computational cost of linearization, which only needs to compute the gradient of the linearized loss.
3 Efficient Utility- Preservine Machine Unlearning
This section first presents the formulation of the Utility- Srettt-pres Unlearning Problem (UPUP). Then, introduce gradient method for solving UPUP, which is equivalent to explicit unilateral gradient surgery. To address the additional computational cost brought by gradient surgery, we propose a method of implicit efficient gradient surgery, efficient approximation for solving UPUP (Algorithm T. Finally, we provide a theoretical analysis of the Pareto optimality/statinnayy.
3.1 Utility-Preserving Unlearning Problem
At iteration 4, perform the update 0.+1 0, a; where d, is the update direction, and define the improvement the retaining and unlearning objectives as follows
(a,, d,) = (0,) (0 Tu (ar, de) = (O,) e., (0.+1)-
To achieve utility-present preserving unlearning, aim to seek direction d, to control the degradation of the local retaining target r, during the constrained optimization process while maximizing the optimization of the unlearning objective. Mathematically, this can be expressed as:
1
ru rular, de) - ||d.1|
max d:
at
(2)
1
-r, s.t.
rr (as, de) 2 -
0t
where at is the stepsize, and ||de the regularization to avoid unbounded solutions. Here, E 2 is the tolerance of the degradation for the retaining target that we aim to preserve, and the constraint Tr (a,, de) - ensures that the utility performance drop is controllable.
3.2 Explicit Unilateral Gradient Surgery
Since stepsize a₂ is usually small, by the first-order Taylor approximation, we know that ry rr de) Rs aa Vl,(O.) di, rula,, de) R az (l.) d.
Problem can be approximated by
max (l,( d, 2 |d
d.
S.L. (4,(0,) d, 2 EEt-
Problem 3 aims to control the dot product of d, and Ve, to be greater than -E,, thereby ensuring that the degradation of f, less than Et, while simultaneously maximizing the dot product of d, and Vl,, to achieve more effective unlearning.
Proposition 3.1. The dual objective of Problem is
min (x,).)
+ +XV, AeSt.
(4)
X,20
Problem 4 has closed form solution, and the desired direction d; can be solved

---

Vlu
Vlu
Vly
V1,
conflicting
non-conflicting
(a)
(b)
Figure 2: Illustration of Unilateral Gradient Surgery. The computation of the update direction is divided into two scenarios: (a) the gradient of the unlearning objective conflicts with the gradient of the retaining objective, that is, the angle between them greater than 90 degrees, the desired direction obtained by removing the projection (to the retaining gradient) component from the unlearning gradient; (b) If the gradient of the unlearning objective does conflict with the gradient of the retaining objective, then the unlearning gradient itself is the desired direction.
d
V4
Vlu
VL
(b)
Figure 3: Illustration of the role of et- (a) When & = 0, if the retaining gradient and the unlearning gradient are opposite direction, the update direction d = O, which point optimization halts and no further progress can be made in optimizing the unlearning objective. direction d f O, and optimization of the unlearning objective proceeds under the condition that does not unduly affect the retaining objective.
(b) When e, 3 0, the update
Proposition 3.2. The closed form solution of Problem 3
dī {
(0) + X, (o,),
if > 0 if S 0
(5)
(l,(
where
X Vl.(O.) (O.
(6)
(0.)l
Due to space limit, we defer the derivation details to the Appendix & It is worth noting that when E, O, the direction of dī is equivalent to performing unilateral gradient surgery on the gradient of the unlearning objective, Vl, (0,). As illustrated in Figure 2 when Vl. (+,) is in conflict with the gradient of the retaining objective, (l.().). the direction di obtained by eliminating the conflicting component of Ve, along (O.). When (e, and (e,(0,) are not in conflict, then d; (O.), and directly optimizing the unlearning objective will not have a negative impact on the retaining objective.
When E, * 0, an error tolerance introduced, gradient surgery only performed when the conflict between the unlearning and the retaining gradients is sufficiently large, that is, when (0, (l,( -,,. This threshold controls the priority during the optimization process for the unlearning objective. Specifically, when the angle between the retaining gradient and the unlearning gradient is 180 degrees as illustrated in Figure if e, = O, the optimization process stops and the unlcarning objective can not be further optimized. In contrast, when E, * 0, certain degree of optimization for the unlearning objective is still guaranteed.
Remark 3.3. We can demonstrate that by imposing local constraints on the retaining objective at each training step, the decline in utility throughout the entire optimization process be controlled
(,,)) - (, SO
Σ E
Proof details are deferred to the Appendix C.4 This illustrates that the increase the retaining objec- tive is controlled by E(= E20g. This indicates that we can achieve the goal of adjusting the hyperparameters.
by
5

---

3.3 Implicit Efficient Gradient Surgery
Although unilateral gradient surgery can ensure that the optimization results meet the expected urility-preserving criteria, the computational cost in the optimization process is double that of the linear weighting method due to the need to calculate gradients for both the unlearning and retaining objectives separately. This can lead to inefficiency in optimization, which contradicts the high- efficiency and low-cost goals that unlearning aims to achieve. To tackle this challenge, we propose an efficient approximate solution to Problem in in the following section, which requires only single step of backpropagation, making the computational cost equivalent that the linear weighting method.
Since the solution to Problem requires knowledge of the gradients for both the unlearning and retaining objectives, we consider using a gradient descent approximation to solve for A₁:
Ac+1 = d V (A))).
However, the gradient of (Ae) still necessitates information about the gradients of the unlearning and retaining objectives. Therefore, we consider approximating the gradient V (2x)) using first-order Taylor's approximation:
V VA, ((A)) (l. (Vl.(0.) A. +
= (e.( d, + &t
1 1 (0.) (0.+1)) + e
an
Consequently, we have an approximate method for solving X without backpropagation:
A A 1,0 1 where 5; (0,) - e,(01+1)) t
(8)
ai
We provide the complete Efficient Seility-Preserving Machine Unlearning (EUPMU) algorithm (Algorithm[] [ in Appendix B The algorithm first computes the approximated weight X by Eq/8with with no backpropagation of the loss function, and then uses X₃ to compute the update direction dę by one backpropagation of the composite loss e. (0.) + (O,). Finally, the model parameter is updated by one gradient step with dg. This process does not require to compute (l,( and (0,)) separately like traditional multi-objective method, and hence save half of the computational cost.
f'stter version of EUPMU. We propose EUPMU-fast as a lightweight variant of EUPMU that removes the second retain-loss recomputation on the same batch. Concretely, whereas EUPMU estimates the retain-loss change step as (t) e, (0,; B - (0c+0 B("), which requires an extra forward pass at dz+1 on (!)) EUPMU-fast uses a stochastic proxy based on consecutive retain batches:
A R 6-(0+11;8/+1)
(0,;;; B(1))
so it incurs additional forward pass. This reduces wall-clock per step (see RTE) but be less stable due to between-batch variation; in our runs, it sometimes underperforms EUPMU, though remains competitive while being slightly faster.
3.4 Theoretical Analysis
Our primary concern is whether X, in Algorithm | can approximate the property of the optimal solution >; which determines whether efficient gradient surgery can achieve the goal of utility- preservation. Therefore, we present the following theorem to elucidate this result.
Theorem 3.4 (Approximate X*). Suppose retaining objective and unlearning objective are both (i) G-Smooth; (ii) L-Lipschitz. At training step i, setting Et < O1),, B./a; O(1/2133 and Σi (1),, we have
- (0040)500///00 131/213)
(9)

---

Table 1: Performance of class-wise forgetting on Imagenette using SD. The best performance is highlighted in bold.
SalUn
ESD
FMN
EUPMU
Forget.Clan Class
UA
FID
UA
FID
UA
FID
UA
FID
Tench
0.00
0.94
0.00
1.18
56.20
0.86
0.00
0.93
EnglishSpringer
0.00
0.79
0.00
0.98
71.40
1.24
0.00
1.52
CassettePlayer
0.20
1.59
3,40
1.75
11.20
1.02
0.00
0.97
ChainSaw
0.00
1.07
0.00
1.55
50.80
0.88
0.00
0.99
Church
0.40
0.99
2.60
1.88
75.60
1.66
0.00
0.87
FrenchHorn
0.00
1.44
0.40
1.15
54.40
1.88
0.00
0.83
Garbage Truck
0.00
1.63
0.20
2.38
58.00
1.10
0.00
1.06
GasPump
0.00
0.81
1.00
2.03
23.60
1.36
0.00
1.04
GolfBall
1.20
1.89
3.20
0.85
83.80
1.12
0.00
1.28
Parachute
0.00
1.06
0.00
1.54
64.80
2.22
0.00
0.79
Average 0.18
1.22
1.09
1.46
59.76
1.27
0.00
1.03
Theorem 3.4 demonstrates that as the training step increases, (X,) gradually converges to L (A()), indicating that when é sufficiently large, A can approximate the condition for utility-reeservation..
next focus on whether Algorithm can converge to Pareto optimal solution, which would indicate whether the algorithm fully optimize the unlearning objective under
Theorem 3.5 (Pareto Optimality). Suppose retaining objective and unlearning objective are both (i) convex with parameter 0; (ii) bounded by B; (ii)) I.-Lipschitz; (iv) ||0,]| is bounded by B for t 1,...,T. T. At training step setting a; = S ((1/G), Ei=o(t (i + 1)8; S O and Ei 0(1), there exist composite loss C(0) (,, + &r, (), (Pas Pr) E A2 such that C(O,) - min (()) < 0(1l))
(10)
Theorem 3.5 establishes the convergence in the convex case, confirming that Algorithm I converges to Pareto optimal solution, and the convergence order matches that of the current state-of-the-arr first-order MOO algorithms.
Remark 3.6. More specifically, combining Equation 7 and Theorem 3.5
we can bound the optimality of the unlearning objective by (0) (,(6') S (1/t), where maxg (0), s.t. (0) (On) x0 O E E,az) It shows that EUPMU converges to the solution with optimal unlearning objective under the constraint of a slight deterioration for retaining objective. This demonstrates that the unlearning objective is sufficiently optimized.
Theorem 3.7 (Pareto Stationary). Suppose retaining objective and unlearning objective are both (i) G-Smooth; (ii) bounded by B; (iii) L-Lipxchitz. At training step 1, setting a; S O(I/G), Ei 1)8; O(1), and Ei=re O(1), have
M| Vl.(O.) t Ve, 50 6 L (Ma.ttr)
min
min
(11)
AA2
Theorem 3.7 elucidates the convergence in the non-convex scenario, demonstrating that Algorithm capable converging to a Pareto stationary point, with a convergence order that remains on par with the current best first-order multi-objective optimization algorithms. This indicates that the algorithm theoretically ensures sufficient and efficient optimization even non-convex situations. We defer all the proof details to the Appendix C.5
C.6
C.8
Experiments
In this section, we present empirical assessments of our proposed approach through experiments on image generation tasks, and benchmark its performance against number of contemporary MU baselines. We leave detailed setups and results well as additional experiments in Appendix D
7

---

Table 2: Quantitative results of instance unlearning and artist style unlearning. The best-performing results are highlighted in bold, and the second-best results are underlined.
Model
Snoopy CA
Mickey
Spongehob CA
Van Gogh
Picusso
Rembrandi
CS
FID
CS
FID
CS
CA
vD.4
74.48
99.36
72.43
97.62
73.06 98.50
73,20
94.75
69.01
90.74
71.65
95.73
Frasing
Snoopy
Erasing Van
Gogh
CS]
CA
FID
CS
CA
FID
CS
CAT
RID..
CSL
CA
FID
CS CA
FID
CS
CA
FID
ESD
49.27
35,38
139.47
58.48
65.00
112.56
62.06
82.25
103.04
51.85
39.25
179.17
63.79
76.50
79.61
65.49
78.94
91.83
ConAbl
57.32
80.88
139.31
69.43
94.38
56.22
70.60
97.00
61.56
57.40
34.00
167.07
65.65
80.83
57.85
68.66
91.95
78.38
SPM
54.82
15.38
111 42
71.89
97.5
30.16
72.79
98.12
42.1
51.7
32.25
198.43
68.47
89.58
23.64
70.83
94.22
41.51
RUPMU
44,9
25.62
167.66
72.74
97.5
29.2
72.52
98.88
40.2
45.01
33.0
228.75
68.69
89.59
22.96
71.01
95.97
39,96
Liasing
Snoupy and Mickey
Frasing Picasso
CS
CA
FID
PTD
CS
CA
HDD.
FID
PI
CS
CA
FTD
LSD
48.99
34.88
29
47.79
23:75
16705
58.03
68.88
11
70.43
85.25
104.56
60.60
51.72
180.50
70.61
93.13
91.70
ConAbl
60.85
94, 50
119
63.46
87.12
108.99
70.06
97.38
66.82
68
83
106.20
58.92
51.21
145 65
70.76
94.88
09.09
54.18
132.08
72.37
73.27
94.75
24.45
258.91
71.5
96.88
EUPMU
46.85
43.25
161.11
44.55
21.0
196.59
71.3
98.63
44.84
73.24
94.75
23.41
45.42
40.49
273.38
71.54
96.37
26.98
Erasing Snoopy,
Mickey
Spongehoh
Rembrandt
CS]
CA
F1D
CA
L RDD
CS
CA
FID
CS
CA
FID
CS 1
CA
FID
CS
CA
FID
ESD
18.6
34,88
147.39
16.91
20.12
173.5
45:22
11.38
192.87
66.68
73.00
87.30
68.97
88.18
79.11
11.33
3,97
206.39
60.48
95.62
84.77
130.92
SPM
51.17
74.25
114.69
53.76
64.25
131.28
52.15
63.88
154.65
72.81
93.75
30.06
68.83
85.73
17.8
32.25
0.22
269.27
EUPMU
47.01
43,5
158.63
44.24
18.75
200.24
42,01
5.37
213.23
73.01
94.5
26.12
69.12
90.49
14.99
27.59
0.48
274.82
ISD
CanAMi
SPM
Our
whwee Sowcer, de 40 Goglls tike
sinonsiii wirr matille vamera abat foom showe rbowdng entekey monoe toithgg pabre trurt mending the sigrofaa baakiing the wo ff offeysssl, will solorfi light
mil Uf framwing cnttorrs futeoon light and shatent and Avousto use of collee resibiiove of Carorogloy pateslege
15D
CocAcl
Uzre
They of ligll and Jussan ie Ronicuedr' fooe Nige Horck
4 Af portaak, Calur angloragginng the foos, Precsss Wive
Anewyy
rkaden civeast city Reedboader dgt gailig skevye
idones Apanye de
Car with lage fonng os the mpleg Moniel
Figure 4: Visualization of instance & style unlearning. Concepts that have been unlearned indicated in red.
4.1 Experiment Setups
Implementation Details. We focus on DDPM [1] and SD (55) models to prevent the generation of specific object classes, and the experiments are conducted on CIFAR-10 and Imagenette [3) respectively. We also consider concept-wing forgetting SD to erase instance & style concepts and
NSFW (not safe for work) content. All numerical results are the mean value over 5 independent trials. All experiments are carried out on two A100 GPUs
Baselines. Our experiments encompas baselines, including saliency unlearning (SalUn) 18] crased stable diffusion (ESD) (22] forget-me-not (FMN) 77 and concept ablation (ConAbl)
Evaluation Metrics. Unlearning Accuracy (UA), CLIP Score (CS) (2), CLIP Accuracy (CA), FID 30 and Runtime Efficiency (RTE) are utilized to measure the unlearning performance in concept-wise forgetting tasks. UA employs an external classifier, which is finetuned to classify the Cifar- 10 dataset, confirm the absence of the forgetting concept in generated images. The CLIP Score calculates the similarity between the image and the prompt. CLIP Accuracy is determined by performing binary classification distinguish between the target and the anchor concepts. RTE is the relative estimated computation time of applying an MU method.
8

---

Table NSFW Content Removal Statistics. We follow the i2p prompts [56] set that contains 4709 samples, and present the number of NSFW contents generated by models before and after unlearning. The unlearning methods include ESD, FMN, SalUn and ours. The best performance is highlighted in
bold.
Category
SD
FMN
ESD
SalUn
Ours
Male genitalia
54
11
17
Male breast
244
51
39
4
Female genitalia
28
10
10
Female breast
225
43
30
4
3
Buttocks
57
12
Total
608
129
108
13
6
Methods
P4
P5
P6
PH
P9
PI0
SD
ESD
SaltIn
Figure Examples of generated images using SDs w/ and w/o MU. Each column represents generated images with the same prompt (denoted by P.) and seed.
4.2 Experiment Results
Class-wise Forgetting in Image Generation.
Table 5: Performance of class- wise forgetting on CIFAR-10 10 with classifier-free guidance DDPM. The best performance is in bold.
Table presents the numerical results for the unlearning the cat- egory 'Airplane' in DDPM. In the numerical results, our approach shows a significantly better FID score compared to ESD and SalUn, while the UA score is consistent with other methods, demonstrat- ing the utility preservation efficacy of EUPMU, which achieves a superior tradcoff in this task. Table T also displays numerical out- comes for SD unlearning across different categories, where EUPMU demonstrates notable improvements in both UA and FID metrics, further confirming the capability of the proposed method to address the unleeming tradeoff issue.
Method Retrain
UA
FID
RTE
1.80
20.50
ESD
0.00
47.57
1.6
SalUn
0.74
24.53
6.05
BUPMU
0.78 22.57 1.38
Instance & Style Forgetting in Image Generation. selected several representative concepts to verify the performance of unlearning instances and styles, as detailed in Table In both instance and style forgetting tasks, our approach nearly outperforms other baselines in all metrics of CS, CA and FID, demonstrating that successfully unlearns specific instances and styles while excelling in preserving non-target concepts. Visualization is shown in Figure 4 we observe that EUPMU can precisely erase target concepts without negative impact model capability. See Appx. Figure 6 for a 3D Pareto analysis over (CS-FF Forget, CA-Forget, (eD-Retain), where integrating EUPMU into both ConAbl and SPM significantly improve the Pareto front to all baselines.
NSFW Forgetting in Image Generation. Table 3 and Figure 5 demonstrate the effect of using Machine Unlearning (MU) to forget the illicit concept of nudity. From Table we observe that our method generates the least number NSFW images, proving that HUPMU more effectively erases NSFW content than other baselines. From Figure find that the quality of the generated images

---

Table 4: Ablation of MOO Methods applied to Random Labeling for Class-wise Data Forgetting(102) of unlearning Resnet 18 for Cifar 10 classification. The best performance is highlighted in bold. We define the average score (Avg.score) to be (UA+RA+TA+MIA)/4
Methods
Description
UA
RA
TA
MIA
Avg. score
RTE
Retrain
Full Retraining
100.00
100.00
94.68
100.00
98.67
Lincarization
Pure Lincarization
98.44
96.88
90.97
100.00
96.57
15.7
FAMO
Efficient MGDA
98.68
98.44
92.56
100.00
97.48
18.7
PCGrad
Confined Gradient Surgery
98.71
98.64
92.65
100.00
97.50
27.7
UNGrad
Unilateral Gradient Surgery
99.53
99.25
93.70
100.00
98.12
27.7
EUPMU
Efficient Unilateral Gradient Surgery
99.64
99.69
94.29
100.00
98.40
18.4
EUPMU-fast
Fast EUPMU variant
98.18
99.83
94.36
100.00
98.09
17.9
is significantly superior to other methods. This indicates that our approach can effectively achieve concept unlearning without compromising the model's generative capabilities.
4.3 Ablation Study
T the results of our ablation study. We use Unlearning Accuracy (UA) and Membership Inference Attack (MIA) for unlearning efficacy, Remaining Accuracy (RA) and Testing Accuracy (TA) for classifier fidelity. We compared representative MOO methods, FAMO [55 and PCGrad [69]. FAMO is an efficient approximation of the MGDA algorithm, while PCGrad represents the gradient surgery approach. This comparison aims to demonstrate that pure MOO algorithms cannot resolve the unnlity-unleariin challenge mentioned in the background. Additionally, we contrasted our results with UNGrad, the explicit unilateral gradient surgery method introduced in previous section, to assess whether the implicit efficient gradient surgery in EUPMU can achieve the effectiveness of the exact method and enhance algorithmic efficiency.
MOO can not solve we observe consistent improvement across all metrics after employing MOO methods. However, the optimization for UA not as pronounced. This is attributed the MOO methods' insufficient optimization of the unlearning goal. Comparing RL+UNGrad with RL+EUPMU, we find that compared to MOO methods, there is sufficient optimization in MIA, indicating that the UPUP modeling approach is better suited for unlearning problems.
tradeoff. Comparing RL with RL+FAMO and RL+PCGrad,
Implicit gradient surgery is efficient. Comparing EL+EUPMU with RL+UNGrad and RL+PCGrad, we find a significant reduction in Runtime Efficiency (RTE) of EUPMU, demontrating the efficiency, At the same time, since it only requires one backward pass, RTE is essentially consistent with the linear weighting method. EUPMU-fast further decreases RTE with removing the extra retain loss forward pass and uses next batch retain loss measure change of retain loss. However the result might be not as good as EUPMU's result, due to the randomness and difference between batches.
Implicit gradient surgery may be more effective than explicit gradient surgery. Comparing RL+EUPMU with RLUUNGrad, we observe a significant improvement in every metric. This is attributed to the fact that under the stochastic gradient training in deep learning, the method of approximating solutions for composite weights À is more stable than the precise solution, thereby achieving better results [76).
5 Conclusion
This paper investigates the core issue of the utility-unlearning tradeoff in machine unlearning, highlighting that existing multi-objective methods cannot be directly applied to unlcarning scenarios due to inconsistencies in modeling with unlcarning goals and issues with algorithmic efficiency. Therefore, this paper first establishes the utility-peeerrving unlearning problem and proposes bradient-based optimization algorithm to solve it, proving its equivalence to unilateral gradient surgery. Subsequently, an efficient implicit gradient surgery method introduced accelerate the efficiency of gradient surgery and avoid introducing additional computational costs. Theoretically, we analyze that the algorithm can converge to Pareto optimal/stationary while maintaining utility, indicating that the algorithm can achieve an optimal tradeoff. Empirically, experimental results validate that the algorithm can fulfill the purpose of utility preservation while sufficiently optimizing the unlearning effect, further corroborating the theoretical findings.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation of China (62401327). We would like to thank Yongliang Wu for his advice on the experiments.
References
[1] Seohui Bae. Seoyoon Kim, Hyemin Jung, and Woohyung Lim. Gradient surgery for one-shot unlearning on generative model, 2023.
[2] Romain Beaumont. Clip retrieval: Fasily compute clip embeddings and build a clip retrieval system with them. https: http://ttttuuu..cc wwt. http://oooooom000////----eeeeeee
2022.
[3] Alexander Becker and Thomas Liebig. Evaluating machine unlearning via epistemic uncertainty. arXiv preprint ArXiv:2208. 10836, 2022.
[4] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
[5] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2023.
[6] Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7766-7775, 2023.
[7] Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 9939911311111 2022.
[8] Zhao Chen, Vijay Badrinarayanan, Chen-Yo Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803 PMLR, 2018
(9] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039-2050, 2020.
[10] Jiali Cheng, George Dasoulas, Huan He, Chirag Agarwal, and Marinka Zitnik. Gnndelete: A general strategy for unlearning in graph neural networks. arXiv preprint arXiv:230..14000 2023.
[11] Eli Chien, Chao Pan, and Olgica Milenkovic. Certified graph unlearning. arXiv preprint arXiv:2206.09140. 2022.
[12] Ana Luísa Custódio, JF Aguilar Madeira, A Ismael F Vaz, and Luis Nunes Vicente. Direct multisearch for multiobjective optimization. SIAM Journal on Optimization, 2103):1 2011.
[13] Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in Neural Information Processing Systems, 32, 2019.
[14] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective opti- mization. Comptes Rendus Mathematique, 350(5-6):313-31 2012.
[15] Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In International Conference on Machine Learning, pages 2658-2667, 2020.
[16] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Annual international conference on the theory and applications of cryptographic techniques, pages 486-503. Springer, 2006.
11

---

[17] Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.
[18) Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprini arXiv:2310. /2508, 2023.
[19] Heshan Devaka Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objectteo learning: A provably convergent approach. In The Eleventh International Conference on Learning Representations, 2023.
[20] Jörg Fliege, A Ismael F Vaz, and Luís Nunes Vicente. Complexity of gradient descent for multiobjective optimization. Oplimization Methods and Software, 34(5):949-959, 2019.
[21] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures, In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
[22] Rohit Gandikota, Joanna Materzyúska, Jaden Fiotto-Kaufi Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision, 2023.
[23] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
[24] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304-9312, 2020.
[25] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-11224 2021.
[26] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint 1eXiv:/911.00000 2019.
[27] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. Proceedings of the European conference on computer vision (ECCV), pages 270-287, 2018
[28] Yifei He, Shiji Zhou, Guojun Zhang, Hyokun Yun, Yi Xu, Belinda Zeng, Trishul Chilimbi, and Han Zhao. Robust multi-task learning with excess risks, In International Conference on Machine Learning (ICML), 2024.
[29] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.087/18 2021.
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[31] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. /2598, 2022.
[32] Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4819-4828, 2024.
[33] Jeremy Howard and Sylvain Gugger.
Fastai: A layered api for deep learning. Information.
1 11(2):108 2020.
[34] Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization multi-task learning: theoretical perspective. Advances Neural Information Processing Systems, 36, 2024.
12

---

[35] Mark He Huang, Lin Geng Foo, and Jun Liu. Learning to unlearn for robust machine unlearning. arXiv preprint arXiv:2407.10494, 2024.
[36] Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, and Karthik Sridharan. Online optimization: Competing with dynamic comparators. In Artificial Intelligence and Statistics, pages 398-406. PMLR, 2015.
[37] Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. arXiv preprini arXiv:2103.02666 1000000261,2222 2021.
[38] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934 2023.
[39] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
[40] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 7482-7491, 2018.
[41] Nupur Kumari, Bingliang Zhang, Sheng-Yo Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22691-22703 2023.
[42] Yunwen Lei, Ting Hu, Guiying Li, and Ke Tang. Stochastic gradient descent for nonconvex learning without bounded gradient assumptions. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[43] Shen Lin, Xiaoyu Zhang, Willy Susilo, Xiaofeng Chen, and Jun Liu. Gdr-gma: Machine unlearning via direction-reetified and magnitdee-adjusted gradients. In Proceedings of the 32nd ACM International Conference Multimedia, MM '24, page 9087-9095, New York, NY, USA, 2024. Association for Computing Machinery.
[44] Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization, 2023.
[45] Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization. Advances in Neural Information Processing Systems, 36, 2024.
[46] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890 2021.
[47] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li. Kush R Varshney, al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.0878)) 2024.
[48] Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to be forgotten in federated learning: An efficient realization with rapid retraining, arXiv preprint arXiv:2203.07320 2022.
[49] Mengyao Lyu. Yuhong Yang. Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue. Jungong Han, and Guiguang Ding. One-dimensional adapter rule them all: Concepts, diffusion models and erasing applications. arXiv preprint 23..::23/2.161455. 2023.
[50] Pamela Mishkin, Lama Ahmad, Miles Brundage, Gretchen Krueger, and Girish Sastry. Dall-
2 preview-risks and limitations. Noudettu, 28:2022, 2022.
[51] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerd, Descent-to-delete: tradient-based methods for machine unlearning. In Algorithmic Learning Theory, pages 931-962, PMIR, 2021.
13

---

[52] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee- ChennC Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint (rXv:22009002999 2022.
[53] Youyang Qu, Xin Yuan, Ming Ding, Wei Ni, Thierry Rakotoarivelo, and David Smith. Learn to unlearn: A survey on machine unlearning, 2023.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10699 2022.
[56] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522-22531, 2023.
[57] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34: 14:18875-18086 2021.
[58] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objectivv optimization. Advances in neural information processing systems, 31, 2018.
[59] Masahiro Suzuki and Yutaka Matsuo. A survey of multimodal deep generative models. Advanced Robotics, 36(5-6)2261-27) 2022.
[60] Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[61] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303- 319. IEEE, 2022.
[62] Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126-4142. PMLR, 2021.
[63] Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. Federated unlearning via dids-diicrimiiaaive pruning. In Proceedings of the ACM Web Conference 2022, pages 622-632, 2022.
[64] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. arXiv preprint arXiv:2 2021.
[65] Leijie Wu, Song Guo, Junxiao Wang, Zicong Hong, Jie Zhang, and Yaohong Ding. Federated unlearning: Guarantee the right of clients to forget. IEEE Network, 36(5): 2022.
[66] Yongliang Wu, Shiji Zhou, Mingzhuo Yang. Lianzhe Wang, Wenbo Zhu, Heng Chang, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. arXiv preprint 2rXiv:2405.1555 15304, 2024.
[67] Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In International Conference Machine Learning, pages 449-457. PMLR, 2016.
[68] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning, arXiv preprint arXiv:2310.10633 2023.
[69] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 1455824-5836.. 2020.
14

---

[70] Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.1759/ 2023.
[71] Han Zhao, Jianfeng Chi, Yuan Tian, and Geoffrey Gordon. Trade-off's and guarantees of adversarial representation learning for information obfuscation. Advances in Neural Information Processing Systems, 1119485-94999 2020.
[72] Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffss in invariant representation learning. Journal of machine learning research, 23(340):1-49, 2022.
[73] Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou. Bandit convex optimization in non-stationary environments. Journal of Machine Learning Research, 22(125))1-45, 2021.
[74] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[75] Shiji Zhou, Lianzhe Wang, Jiangnan Ye, Yongliang Wu, and Heng Chang. On the limitations and prospects of machine unlearning for generative ai. arXiv preprint arXiv:2408 00377, 2024.
[76] Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie Gu, and Wenwu Zhu. On the convergence of stochastic multi-objective gradient manipulation and beyond. Advances in Neural Information Processing Systems, 35:38103-38111 2022.
[77] Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: comparative case study and the strength pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257-274 1999.
15

---

A Related Work
Our primary focus is on addressing pivotal issue within MU - unlearning- retaiinng tradeoff. We will first compare our approach with existing MU methods, elucidating why current techniques fall short in resolving the targeted issues. Subsequently, we introduce the multi-objective optimization approach, which may offer solutions to the tradeoff problem. We will discuss the reasons why these methods cannot be directly applied to MU and propose our novel strategies to effectively tackle these complex problems.
Machine Unlearning (MU) aims to refine machine learning models by eradicating the impact of
certain data points or classes, primarily avert potential privacy violations post-training (23 [1 62
(57. The ideal of complete unlearning, akin to retraining from scratch, is computationally prohibitive despite its theoretical merits. To mitigate this, research has ventured into probabilistic techniques
such as Differential Privacy (DP) [23 26
51 62 57]. Yet, these techniques encounter limitations
that impede their efficacy, notably in thwarting membership inference attacks [16] 25). Consequently,
there is a pivot towards crafting more potent and economical MU strategies [24] 3 61 38
6 64] The
reach of MU has extended into federated learning [3 48 65] and graph neural networks
amplifying its utility in diverse data ecosystems. Nonetheless, current methodologies grapple with balancing unlearning effectiveness and model utility, alongside the adaptability of MU methods across varied scenarios. A pertinent work, SalUn [18], hamnsssss gradient information for parameter selection. However, most of current methods neglect the critical role of dispersing selected parameters across the network for thorough unlearning and fails to tackle the gradient conflict issue, a pivotal
aspect of optimizing unlearning processes. is noteworthy that some works [32 35| 666 also adopt
the technique of explicit unilateral gradient surgery. However, they are based heuristic methods, while our paper provides the original optimization framework and theoretical support, which is quite different in terms of the principle origin. Also, the computational cost of explicit unilateral gradient surgery is almost double than EUPMU.
Concept erasure in diffusion models. Beyond discriminative MU, text-to-image diffusion erasure has emerged as a parallel line tackling the removal of artists/styles/issaacess SPM introduces a one-dimensional, aldg-and-play adapter with latent anchoring and input-dependent permeability for non-invasive multi-concepr erasure [49]. ConAbl (Concept Ablation) matches the target distribution to an anchor concept to prevent generation of a specific style/instance while preserving related concepts [4]]. ESD (Erasing Concepts from Diffusion Models) fine-ture tun-tunes diffusion weights with negative guidance teacher to permanently remove concept (22). Notably, SPM and ConAbl both optimize a forget loss and an optional retain loss via linear weighting. Our EUPMU view suggests recasting such objectives under explicit utility-loss constraint (retain budget) while optimizing the unlearning objective-i.e. a constrained alternative to linear scalarization iillaiizatio-whiill whhic can be implemented by our implicit unilateral surgery in single backprop and could directly boost this family of methods.
Other gradient-operation MU. Methods like GDR-GMA (using direction-rectified, magnitude- adjusted gradients) 43) and Learn to Unlearn 53 (manipulate gradients through projecting gradients away from pre-computed Core Gradient Space (CGS) to mitigate confficts) both use gradient related operations in MU. These are largely heuristic gradient corrections or projections, whereas our approach derives the exact surgery from optimization principle (the Problem) and exposes a user-specified utility budget for proactive, interpretable control. Our methods also could avoid extra calculation coming from gradient operation through efficient implicit gradient operation.
Unlearning
Multi-Objectin Optimization (MOO) have been developed address the concurrent learning of multiple tasks through gradient modulation. A common approach in these methods involves dynamically re-weighting objectives based on uncertainty metrics [40], gradient magnitudes [8] and the complexity of training tasks 27] The structured design and stable training of Multi- -bjeit-Oeeeetve Optimization (MOO) strategies have attracted significant interest. For example, [58] framed Multi- Task Learning (MTL) within an MOO context, proposing an adaptation of the Multiple Gradient Descent Algorithm (MGDA) for the task. Several techniques have been introduced to resolve gradient conflicts. Notably, (69 developed PCGrad, which aligns each task's 's gradient within the norm plane of the others. GradDrop reduces conflict by stochastically dropping conflicting gradients (9) RotoGrad resolves conflicts through gradient rotation (37], and [46] introduced CAGrad, which
16

---

Algorithm Efficient Unility-rreserving Machine Unlearning (EUPMU)
Input: Initial model parameter On 01 to be pre-trained model, initial retaining parameter Àg learning rate {at, St f error tolerance {er 17 unlearning loss (+),, maximum value for X and retaining loss (-((-)
for 1, ...... T do
2: Update retaining parameter X by Eq. 8 as
8 = au (6-(0)-1) - e, (r()) + El-1
X min D. max 10.1-11 ----10--1
3: Update direction d, Vo. (0.) X,l, (0.))
4: Perform gradient step 0c+1 = 0, a.d.
5: end for
constrains gradients within a localized region around the average gradient direction. These strategies primarily target deterministic scenarios. [19] advanced MoCo as probabilistic counterpart to MGDA, providing a comprehensive convergence and complexity analysis. [44] the efficient method for MGDA, which address the additional computation cost brought by MOO. However, discussed in the main text, the MOO methods fail to directly applied to MU.
B Algorithmic Details
Note that the clipping operator of step 2 in Algorithm Tis due the fact that d 2 0 and the practical need for fear of parameter explosion.
C Missing Proofs
In this section, we present the proof details.
C.1 Description for Assumption
In this paper, we have the following assumptions. Assumption C.1 (L-Lipschitzz.. For the objective function S, there exists constant 7, > 0 such that for v, y Rdd,
llf(z) f(y)l < L|| yl|-
Assumption C.2 (G-smoothness). The objective function f is differentiable and its gradient V is G-Lipschitz continuous, i.c., there exists a constant
such that for all x, y R..
[Vf(x) vf(v))l = - øll-
Assumption C.3 (Function bounded by B). The objective function f is bounded by B, i.c. there exists a constant B > O such that for all æ E RRd
[f(x) < B.
C.2 Proof of Proposition 3.1
Proof. To derive the dual objective of Problem 3 start by constructing the Lagrangian function associated with the primal problem. The Lagrangian can written as:
L(d,, A.) (O. d, |d, > (e+ (e,(0.) d:),
where 2 is the Lagrange multiplier corresponding the constraint in the primal problem.
To find the dual objective, we first need to minimize the Lagrangian with respect to d,. Taking the gradient of L with respect to d, and setting it to zero, have:
Va Ad) (l. + (e,,, d, 0.
17

---

Solving for d,, we obtain:
d, (0,( + (o.).
Substituting this back into the Lagrangian, get the dual function:
(Ae)) A(d;,At)
vlu(0.) (VL, (
1
- [V.), V. A
Expanding and simplifying, the dual objective becomes:
(A.) (.. + (O.) + A
which is the dual objective stated in Proposition 3.1
C.3 Proof of Proposition 3.2
Proof. To find the closed-form solution for the optimal direction d;, we first minimize the dual objective (de) with respect λ₁. Taking the derivative Le()e) with respect to X and setting to zero, we get:
(L, (V A + =
Solving for Å. we obtain:
X; (O,) (e, -4 (0.]l=
Substituting X; back into the expression for di, we find the optimal update direction d,
if x; > 0, di (e.( + (o)), (l.)
if X < 0.
C.4 Demonstration of Remark 3.3
Proof. By the G-smoothness, we have
(,(0.+1) (r.) S (1, + o&G ||d;12
a/G ||d,12
S 0;;; 2
S a;s; (if a is sufficient small).
The last inequality because a; is sufficient small, the the second term is much smaller than the first term, and hence can be ignored in the approximation. By summing up the above from 0 to t 1, we can get
- (,,))))(( $ O
Σ c(a))
We hence proof the Theorem.
Remark. If we set Eis Ero, < E, then the total loss ascent bounded by e, and hence the performance drop be controlled.
18

---

C.5 Proof of Theorem 3.4
Our proof adopts the following approach. The process of solving for À equivalent to a gradient update with a dynamic function, and the essence of the theorem is to bound the dynamic regret. Therefore, we intend draw on results from online gradient descent (OGD) for dynamic regret to prove the theorem, where the crucial requirement of using OGD property is whether the total functional variation controllable. Hence, we initially present lemma to demonstrate this.
Lemma C.4. Under the same assumption as Theorem 3.4 variation for dual function
we have the bound for the total functional
sup (1) -x(AA]
1=0
s <D+ D
|i41 Ei
1=0
1=0
Proof. By the definition of (A),, we have
(++1 (A) - L,(1 =
A64+1
1 (V,)((,) + +
(Veu(0 Vl. ( +
(VL,(0.+1) - (O, -
+ A1E++11 - Eg|
< a G)2 + Alei+1 - =,,
where the last inequality is from the assumption of L-Liischitzz and G-smoothness, we know that
|| (0.41) (l, + (0.) A (0:) S 2 + XL), and
[[V - (VC
(e,)l
< (1] - |IAV, - (0,)(()))
I GG11041 - 0,11 AG| - 0,1| (G-smooth)
(A+ a,,]la,d,d
= aa(A+ + 1)GT|VL.((HH)) + (0+,
S saiAA 1)G(1 A)L (L-Lipschitz)
aalA+ GL.
Hence, we can get
sup L++1 - (,(A)
1
(A + Neii11 Exl)
1=0
< + 1)
Dle141 Ev
1=0
(D + =
a + D
Σ |=:1 53
0
We now finish proof.
We now present the previous results of OGD, which can be later used for the final proof.
19

---

Lemma C,5 (Theorem (6]), Denote the total funtional variation
V Σ sup (++ - f((z)) x
Online Gradient Descent algorithms with stepsize a = 10v (-1/3) running with enjoy the following bound
E
p.s))
With the above results, we now can proof the theorem
Proof. By setting Ei=o aN S 0(1) and Ei=oe: ei S O(1), we know that
Σ sup (++1 Lx S 0(1)
From the algorithm [ with stepsize B./a; (11/1133). Using the result of Lemma C.5 we can get that
we know that the update of À is the process of online gradient descent 667
(L.(A.) - L((1))
0(1
sup (A)()))))()))))333
i=0
((233)
Dividing both side by t, we finally have
1 Σ -4())- 0(1/813) 2=
i
We hence end the proof.
Remark. It should be noted that the presented results pertain the use of the true gradient of L., whereas approximation of the true gradient is employed for the update of A. However, this approximation is two-point estimate of the true gradient, and prior research [73] has demonstrated that the dynamic regret associated with such an approximation is equivalent to that of OGD when using the true gradient. Therefore, for the sake of clarity and to avoid redundancy, we have omitted this detail in the presentation.
C.6 Proof of Theorem 3.5
This convergence proof parallels the approach to obtaining a dynamic regret bound. We adhere to the foundational concept from [76), initially establishing a bound for the static regret, followed by bounding the discrepancy between the static and dynamic regrets. The synthesis of these bounds culminates in the final theorem. Hence, we first present the following Lemma.
Lemma C.6. Under the same assumption as Theorem algorithm 7 enjoys the following bound
3.5
denote Cx, (o) + (0),
Σ (.) min 0
Cx,
2B
1 - -100 - 0- DB.
<
3.(
+ +x) +
2a
1=0
20

---

Proof. By the G-smoothness, we have
agG
(x,(0++1) a,d, VC,, (0,) +
d.
2
G ||d:1l
Cxx(0:) as||del' 1| +
0: |d:11? 2
& Cs,
The last inequality is due to the fact that o;G S 1. By the convexity of both function, know that
(O.) S (x + d; (0; -0))
Plug into the first inequality, we obtain
Cx, S Cx. (0") + d, (0, 0¹) ai
2 |d: ||
(0") 1
20: (100: 0- 00 0" a;d, l|²)
(0") + 20
(10: 1 00++ 0- 13).
For any 0, we have
Cx.,1 - Cx. = (Act1 - (.)
< B,B/oll.
We then can get
Ca -0++1) (0")
-
Cx (0++1) Cx, (11) + (x.4111) Cx, (0')
B.Bloa + 2a;
() 0- 112 - 10:+11 0- 13)
Let a; = a, i = 1. t to be constant, and we have
Cx. (0,)
Σ Cx.
(Ch44 (0:+1) Cx)( + Cx" (0"
1
3,B 8,1 + (1 1-112 100 11)
2a
(Ao- (0))
1
3,Bl +
(1100 0-1 - -100, -0"113) + DB
2a
1
< B
BGL +x) + - 10 - -0111 +DB
2a
The second inequality is by the assumption that A is bounded by D and functions are bounded by B, and the last inequality is by fact that
18,1
(O.) (.((++1)) + eal
(O.) (.(0.+1)) +
< 1 G -0--Il +
Galldil +
< GLG+E Ei.
21

---

The second inequality is by the G-smoothness, and the last inequality is by L-Lipschitz assumption. We thus end the proof.
With the bounded static regret, we also need to introduce the following lemma to bound the gap between static and dynamic regrets.
Lemma C.7. Under the same assumption as Theorem 3.5
algorithm
ennooy the following bound
min 0"
(0))
Σ
min (0)) 1:
<
B
iB. + Ex)
2B
1=1
i=1
Proof.
Denote À = + A; By the optimality of 0", we can know that
Σ Cx. (0) - Σ (2)
= (())(0") + (e)) - Σ (7,)
i=1
C0 Σ Cx. (07)
- (0,)
B Σ A Adl
The first inequality is by the fact that (0) + (0) S Cx(,,) from the optimality of 0", and the
last one is from the bounded function assumption. We next try to upper bound Σi= [ス AsI. and have
~0 -
- A/ 12
{=1
=
ΣΣ
1y A+|+ 7 ΣΣ
12 - Ao
j=i
i
2
11 1+1l
(t 1+ 1 A411 2
1) A
1=1
i=1 1=i
2 U/x A411 52 2
13,10,1 5 Σ (G(G++e)))
1=1
We thus end the proof by
Σ Cx - Σ Cx (07) <
Σ
B
8 (G.. + 6:).
We observe that the gap between static and dynamic regrets is controlled by the stability of X, which the advantage of the approximate algorithm compared with the explicit gradient surgery. We are now ready to present the final proof.
22

---

Proof. Combining the results from Lemma C.6 and Lemma C.7 we can obtain
Σ Cx. (0,)
Cx. (0
t
Σ (O,)
Cx
Cx2 (0))
Cx. (07)
1
Σ + 1)33 +
100 0-1 DB.
20
Setting a, = - O(1/G), Ei (1 1)3, - 01), and Ei= E, S (1)). we get
(o.) Σ (x, S 0(1)
Therefore, finally have
Cx. (0.)
Σ Cx,(,)) - 01/1).
This averaging convergence scheme for can be transformed to the traditional one in Theorem 3.5 by
output the average solution from 0z, 1.... T 13 42 15 This paper leaves out the transforming details.
C.7 Demonstration of Remark 3.6
In the Theorem 3.5 we have demonstrated that EUPMU can converge to the Pareto optimal solution, that is,
C(O,) ((0) 0(1/t)
Then we can get
(n,) (ulll(0 + (0.) - =
By the fact that (0,) 2 10") we have
Pulu (0.) - (uln((") 5 ((1tt).
Combining with the results of the theorem in Remark 3.3 we can know that the converged solution satisfies the condition that the decrease of the retaining loss is controlled by
(0.) - (,) S 0
Σ
Spar
By the fact that (,,),) 2 1, finally get
1
6,(0") - (,00 50
Σ
C.8 Proof of Theorem 3.7
To prove the Theorem, we follow a similar procedure to that multi-objective proofs 20), first establishing local recursive inequality, and then bounding the sum of these inequalities. Lemma C.8. Assume that Ly is G-smooth. Let a; S 1/G. We have
2 (L.( - (.. + 2X,
| 12 < 0g
23

---

Proof. By the G-smoothness, we have
(u(+++1) -n,(0,) S -a;d; (0,(0,) af ||
= =ailldii| G
|| 1| 0, Ve, (0:) d,
$ 2
d a d;
The equation is by the fact that d; = (l,(0,)+ + (e:)) The last inequality is by the parameter setting that a; S 1/G. Hence, dividing both side by as/2,, we have
2 ||d:1? 2 (O,) (0(++))) (A,
Oi
2 (c. (0++1)) + 27,Es.
The last inequality from the closed from solution of X which lead to the fact that when o then
ve,(0)) d. = -V,, (V) +
-V,((O)) (V.E.(O.) (e, (L, - (e.)
[0.)(
vl,(0,) (V.)(0,) (e, (O,) (e,)).)) +
(0,),
Et.
where last equation is because the dot product of (2, and the projected gradient (e,(
Ve, (O,) is zero. We hence prove the Lemma. [0,))
Lemma C.9. Denote that d,-
1+7, d;
Σ
1 <
+
B
+ 2 E
a1
a++1
1=1
Proof. From Lemma C.8
know that
2
||c 2 < (0.) - (.((1)))) + 2N,4,
and hence
2 (l, (u(0.+1)) + 2),,))
|| < (Λ, + 1)2 a4
By the fact that X; 2 0, we then get
1
|d 1| < (0.) - la(0.+1)) + (A: - 1)2 A
2
(Pa (0.) (0,+1)) 28t-
a;
Rearranging the above inequality, we can further obtain
| 12 < ((,))
(0(1+)1)
ai+1
2 2
(
) (0.1)) + 2E;
ai+1
ai
l (0)
l (0.11)
0i+1
2
2
(
B 22
06+1
ai
24

---

By summing the above, we have
2 (.))))
2
d
12
-
(0(41))
04+1
1=1
2
+
+
2
0i4l
0i
1=1
i=1
2
2
ly (01)
l (0+11)
01
04+1
2
+
B
2
E1
a++1
a1
1B+27
Ei.
01
04-1
i=1
We thus end the proof.
Proof. By setting ag get from Lemma C.9 that
I,...,7' to be constant such that a; S 1/G, and Ei=1 9; S O(1), we can
=
Dividing both side by t, we obtain
| S 0(111)
8
Finally, we end the proof by
min l.....
min (e
| (O:) + μ4, (0,)l
min
| <
|ا < 1(1/1113)
D Additional Experimental Details and Results
We provide the experimental details and additional results in this section.
Average Gap (Avg.gap). Unless otherwise stated, all classification tables report the gap to retrain in the parentheses after each metric and Avg.gap in the last column defined as
Avg.gap 1 (IUA UAvetrainl RA RA
TAyetrin + MMI - MIA )).
TA
computed on the same test split as the retrained reference. Lower is better.
Common protocols and configs. To avoid redundancy across 20+ runs, we follow the default hyperparameter templates from prior work and release the configuration files/command in our code repository. For classification, the MIA setup, train/val/test splits, and metric computation follow SalUn. For concept selection in style/instance experiments, the anchor concept protocol matches SPM. We include the search ranges in Appx. D. for reproducibility.
25

---

D.1 Additional training and unlearning settings
MU for image classification For image classification, we use Unlearning Accuracy (UA) and Membership Inference Attack (MIA) for unlearning efficacy, Remaining Accuracy (RA) and Testing Accuracy (TA) for classifier fidelity, and Runtime Efficiency (RTE) for computational efficiency in MU. The RTE is measured by relative overall runtime through the whole unlearning. To report overall performance succinctly, we use the average gap (Avg-gap), defined as the mean absolute difference from fully retrained model across {UA, RA, TA, MIA]. J. lower is better. Our experiments encompass unlearning baselines, including FT [64). RL 24 GA (24.. IU [6]], 4-spars [38. boundary unlearning BS/BE [6], SalUn [18]. and -operittto-opraiionnnnn unlearning GDR- -MRA [3 GDR-GMA uses direction-rectifen and magnitude-adiues gradients to update the retain and unlearning loss.
Training for both FT and RL methods occurs over epochs to search for optimal learning rate within [le-4, 1e-2]. Both GA and GDR-GN - use S-epoch learning rate search within [1e-6, 1e-3]. For IU. the parameter a related to the woodfisher Hessian Inverse approximation is explored between [1, 20]. ---paree involves learning rate search for y within [1c-6, le-4], while keeping constant rate of 0.1. In the BS method, the FGSM step size is set to 0.1. Both BS and BE methods include a 10-epoch learning rate search within [1e-6, le-4]. SalUn and SalUn-soft are trained for 10 epochs with a fixed saliency map ratio of 0.5 and searched for the optimal learning rate with in [0.1, le-4]. Both EUPMU and EUPMU-fast using both unlearning and retain tasks for the optimization target, within [0.01, IJ and then searched for é within [1e-3, 5e-1].
For the ablation experiment, RL and EUPMU remain the same settings. FAMO is searched with weight learning rate from le-4 to 3e-2. PCGrad is searched learning rate within [le-4, 1e-2]. Unilateral Gradient Surgery, also using log loss as the optimization target for both unlearning and retain tasks, under fixed learning rate 6e-5, is searched for e within [le-3, 5c-1]. Here the RTE is measured by one single epoch runtime under different MOO methods.
MU for image generation For DDPM, the forgetting settings are as follows: The batch size is set to 128. For EUPMU, it is trained for 1,000 iterations with learning rate of le-4, an a value of 1e-3, and batch size of 128. The sampling settings include 1,000 timesteps and a conditional scaling of 2.0. B3 is searched within [le-4, 1e-2] and is searched within [le+1, 5e+3]
For SD, the forgetting settings are as follows: For EUPMU, it is trained with the Adam optimizer for 5 epochs at learning rate of le-5. The a value is set 0.01, and the batch size 8. The 3 is set to le-4 and æ is set to le+3 The sampling settings use DDIM with 100 timesteps and a conditional scaling of 7.5.
MU for Instance removing We employ ChatGPT to create 200 prompts {c} containing the anchor concept, such as "Dog". Following a similar process to the style pipeline, we generate 1000 images using the pre-trained diffusion model and obtain the target text prompts } by substituting the word "Dog" with "Snoopy".
MU for Style removing To remove style concept, we employ generic painting styles as an anchor concept in the process of style removal. Firstly, detrreetieeva (2) is used to obtain set of text prompts {c} that are close to the term "painting" in the CLIP latent space. Then, 1000 images are generated from the original model with 200 prompts. The target prompts {c*) are derived by appending the suffix "in the style of [target style]" to (c).
MU for NSFW removing To remove NSFW content, we started by generating 800 images as d, using SD V1.4 with the prompt "a photo of a nude person," and another 800 images d, with the prompt a photo of a person wearing clothes." In the forgetting process, we treated "a photo of a nude person" as the concept to be forgotten and "a photo of a person wearing clothes" as the corrective remain concept. The settings for EUPMU remains identical to SD
D.2 More Details of Metries
To assess the model's s ability to unlearn the target concept and retain the generation capability, we utilize three key metrics: CLIP Score (CS), CLIP Accuracy (CA) [29] and Fréchet inception
26

---

Table Results of class-wis forgetting for various methods of unlearning Resnet 18 for Cifar 10 classification. The result in the table is the mean value over 10 independent trials. The best unlearning performance for each forgetting class highlighted in bold.
Methods
UA
RA
TA
MTA
Avg. Gap
Runtime
Retrain
100
100
94.68
100
FT
98,49 (1.51)
97.09 (2.91)
91.43
100.00 (0,00)
1.92
2.28
RL
98.44 (1.56)
96.88 (3.12)
90.97 (3.71)
100.00 (0.00)
2.10
2.45
GA
98.18 (1.82)
81.47 (18.53)
76.99 (17.69)
97.89 (2.11)
10.04
0.13
IU
98.33 (1.67)
(5.00)
(5.49)
99.42 (0.58)
3.19
3,25
BE
93.80 (6.20)
98.28 (1.72)
92.38 (2.30)
99.58 (0.42)
2.66
0.25
BS
92.24 (7.76)
96.75 (3.25)
91.03 (3.65)
98.82 (1.18)
3.96
0.41
& 2i-sparse
96.47 (3.53)
98.18 (1.82)
92.56 (2.12)
100.00 (0,00)
1.87
2.29
SalUn
98.00 (2.00)
99.91 (0.09)
94.90 (0.22)
(0.00)
0.58
2.46
SalUn-soft
98.96 (1.04)
99.75 (0.25)
94.41 (0.27)
(0.000
0.39
2.50
GDRGMA
95.30 (4,70)
100.00 (0.00)
88.70 (5.98)
100.00 (0.00)
2.67
3.47
EUPMU
99.64 (0.36)
99.69 (0.31)
94.29 (0.39)
100.00 (0.00)
0.27
2.82
EUPMU-fast
98.18 (1.82)
99.83 (0.17)
94.36 (0.32)
100.00 (0.00)
0.58
2.52
Table 7: Results of Random Data Forgetting(10%) for various methods of unlearning Resnet 18 for Cifar 10 classification. The results are the mean value over 10 independent trials. The best performance is highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
5.24
100.00
94.26
12.88
0.00
FT
(2.44).88
98.51 (1.49)
92.28 (1.98)
(5.75)
2.92
RL
3.27 (1.97)
99.89 (0.11)
94.10 (0.16)
23.73 (10.85)
3.27
GA
(2.95)
98.67 (1.33)
92.34 (1.92)
3.78 (9.10)
3.83
IU
0.56 (4.68)
99.51 (0.49)
94.58 (0.32)
1.02 (11.86)
4.34
BE
2.80 (2.44)
97.28 (2.72)
90.96
19.96 (7.08)
3.88
BS
0.71 (4.53)
99.24 (0.76)
93.64 (0.62)
13.64 (0.76)
1.67
i-sparse
3.55 (1.69)
99.17 (0.83)
91.36
9.09 (3.79)
2.30
SalUn
5.02 (0.22)
99.83 (0.17)
93.58 (0.68)
27.38 (14.50)
3.89
SalUn-soft
5.36 (0.12)
99.71 (0.29)
93.29 (0.97)
24.53 (11.65)
3.26
GDRGMA
5.42 (0.18)
99.53 (0.47)
92.21 (2.05)
34.61 (21.73)
6.11
EUPMU
3.71 (1.53)
99.25 (0.75)
93.09 (1.17)
12.89 (0.01)
0.86
EUPMU-fast
3.64 (1.60)
99.54 (0.46)
93.17 (1.09)
13.29 (0.41)
0.89
After generating images with the big artists prompts from ESD 22 for art style unlcarning and templates prompt proposed by CLIP (54), calculates the cosine similarity between the generated image and the target concept embedding(e.g. "A quiet moment in Rembrandt's interior scene" or "Snoopy"). Likewise, CLIP Accuracy evaluates performance on a binary classification task distinguishing between the unlearning and retain concepts for each generated image, based on comparison of Clip Score. In both cases, lower metric values signify more effective concept ablation, while higher metrics indicate better retain of concept. Fréchet inception distance (FID) metric used to evaluate the quality and diversity of image generation. We use it to measure the similarity between the images by original Stable Diffusion model and our models' generated data distributions by comparing the mean and covariance of features extracted from pretrained Inception-và model. A low FID indicates a similar distribution of two groups of image and better generation quality, while high FID refers to better unlearning result.
distance (FID) 30
D.3 Additional Results and Demonstrations
and report pondom-data forgetting results. Numbers in parentheses are the absolute gap to the retrain model for that metric. Across CIFAR-10, CIFAR-100, and Tiny-ImageNeet2000 the "Avg-gap" column quantifies the mean distance between each method and an ideal retrain; smaller is better. Our method yields consistently lower Avg.gap across forgetting ratios, indicating a favorable trade-off between forgetting efficacy and utility retention, in line with algorithmic design and theory.
Random Data Forgetting in Image Classification. Tables 7
27

---

EUPMU+SPM
EUPMU ConAal
SPM
F CeeeCnn bettee
660
Forgle Tomeer Bettal)
Chrhorgel dowent better)
C Forger flower bettor)
Churrrpe dhwen better
(a) SPM
(b) ConAbl
Figure 6: Illustration of Pareto Front Analysis. We conducted comparative analyses by integrating EUPMU with SPM and ConAbl forgetting paradigms: (a) We compare EUPMU+SPM with pure SPM; (b) We compare EUPMU+ConAbl with pure ConAbl.
Table 8: Results of Random Data Forgetting(30%) for various methods of unlearning Resnet 18 for Cifar 10 classification. The results are the mean value over 10 independent trials. The best performance is highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
6.50
99.99
92.68
14.44
0.00
FT
4.93 (1.57)
96.89 (3.10)
90.75 (1.93)
11.10 (3.34)
2.48
RL
5.20 (1.30)
99.68 (0.31)
92.11 (0.57)
33.35 (18.91)
5.27
GA
0.50 (6.00)
99.48 (0.51)
94.64 (1.96)
1.07 (13.37)
5.46
IU
4.99 (1.51)
94.97 (5.02)
(3.94)
(6.80)
4.32
BE
0.59 (5.91)
98.82 (1.17)
93.89 (1.21)
8.19 (6.25)
3.63
BS
0.63 (5.87)
99.46 (0.53)
94.15
7.26 (7.18)
3.76
i-sparee
4.76 (1.74)
97.30 (2.69)
91.26
10.36 (4.08)
2.48
SalUn
(1.71)
99.53 (0.46)
91.89 (0.79)
28.08 (13.64)
4.15
SalUn-soft
4.43 (2.07)
99.62 (0.37)
92.42 (0.26)
25.38 (10.94)
3.41
GDRGMA
6.57 (0.07)
99.53 (0.46)
91.79 (0.89)
38.41 (23.97)
6.35
EUPMU
3.53 (2.97)
98.34 (1.65)
91.80 (0.88)
16.50 (2.06)
1.89
EUPMU-fast
5.01 (1.49)
98.04 (1.95)
91.47 (1.21)
16.19 (1.75)
1.60
Class-wise Forgetting in Image Classification As shown in Table 6 cross all 10 categories' class- wise unlearning, our experiments consistently yielded results that align with the findings reported in the paper. Table 66 further details the performance for class-wise forgetting, where the metrics and the absolute gap are shown and the "Avg. Gap" column represents the average performance difference across unlearned (UA, RA, TA, MIA) and retrained models.
Visualization of Class-wise Forgetting in Image Classification Figure demonstrates the Vi- sualization of Class-wise wise Forgetting in Image Classification. Our analysis reveals that compared with baseline methods, EUPMU effectively enhances the erasure performance for target classes while preserving the capability to generate other classes. This observation corroborated by the quantitative results presented in Table
28

---

Table 9: Results of Random Data F0rgetting50%) for various methods of unlearning Resnet 18 for Cifar 10 classification. The results are the mean value over 10 independent trials. The best performance highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
8.11
100.00
91.24
19.72
0.00
FT
2.84 (5.27)
98.75 (1.25)
91.87 (0.63)
7.35 (12.37)
4.88
RL
5.84 (2.27)
99.39 (0.61)
90.69 (0.55)
43.32 (23.60)
6.76
GA
0.62 (7.49)
99.40 (0.60)
94.49
1.23 (18.49)
7.46
IU
7.57 (0.54)
22.82
85.19))
12.36 (7.36)
5.22
BE
10.69 (2.58)
89.72 (10.28)
82.88 (8.36)
22.72 (3.00)
6.05
BS
10.63 (2.52)
87.43 (12.57)
80.59 (10.65)
22.61 (2.89)
7.16
1.27 (6.84)
93.39 (6.61)
98.69 (7.45)
9.49 (10.23)
7.78
SalUn
5,24 (2.87)
99.15 (0.85)
90.94
36.76 (17.04)
5,26
SalUn-soft
5.67 (2.44)
98.91 (1.09)
90.85 (0.39)
38,30 (18.58)
5.62
GDRGMA
4.46 (3.65)
99.60 (0.40)
91.68 (0.44)
36.51 (16.79)
5.32
EUPMU
4.71 (3,40)
96.96 (3.04)
90.35 (0.89)
19.55 (0.17)
1.88
EUPMU-fast
4.08 (4.03)
98.02 (1.98)
91.11 (0.13)
19.84 (0.12)
1.56
Table 10: Results of Random Data Forgetting(1095 (10%) for various methods of unlearning Resnet 18 for Cifar 100 classification. The results are the mean value over 10 independent trials. The best performance is highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
23.07
99.97
74.43
49.29
0.00
FT
6.07 (17.00)
98.45 (1.52)
71.44 (2.99)
17.56 (31.73)
13.31
RL
24.82 (1.75)
98.92 (1.05)
69.50 (4.93)
71.60 (22.31)
7.51
GA
4.16 (18.91)
96.90 (3.07)
74.37 (0.06)
8.62 (40.67)
15.68
IU
10.58 (12.49)
(9.65)
65.92 (8.51)
17.36 (31.93)
15.64
BE
3.40 (19.67)
96.72
71.58 (2.85)
13.58 (35.71)
15.37
BS
3,23 (19.84)
96.41 (3.56)
72.11 (2.32)
12.61 (36.68)
15.60
fi-sparse
2.40 (20.67)
98.14 (1.83)
75.85 (1.42)
7.31 (41.98)
16.48
SalUn
24.84 (1.77)
99.27 (0.70)
69.57 (4.86)
77.96 (28.67)
9.00
SalUn-soft
24.73 (1.66)
98.73 (1.24)
69.39 (5.04)
70.76 (21.47)
7.35
GDRGMA
56.67 (33.60)
99.63 (0.34)
(5.799)))))
93.71 (44.42)
21.01
EUPMU
9.67 (13.40)
99.60 (0.37)
72.03 (2.40)
51.76 (2.47)
4.66
EUPMU-fast
28.36 (5.29)
98.82 (1.15)
69.50 (4.93)
70.36 (21.07)
8.11
E
Pareto Front Analysis
To quantitatively demonstrate EUPMU's enhanced trade-off balancing capability, we conducted comparative analyses by integrating EUPMU with SPM and ConAbl forgetting paradigms, followed by Pareto superiority evaluations against baseline methods. Specifically, following the experimental configuration for Style Unlearning as described in the main text, we performed style erasure on Van Gogh's 's artistic style while quantifying three performance metrics: (1 & 2) CS-forget and CA-Forget: CS and CA scores for Van Gogh style erasure efficacy, and (3) FID-Retaint FID score for Picasso style preservation. This tripartite evaluation framework provides analytical support for EUPMU'S superior trade-off performance through comparative analysis of these interrelated metrics.
Figure 6 presents comprehensive experimental validation, demonstrating that integration with EUPMÜ enables SPM and ConAbl to achieve consistent improvements across all evaluated metrics, thereby yielding superior trade-off optimization outcomes. This empirical evidence indicates that EUPMU's optimization mechanism can effectively identify Pareto-optimal solutions through its enhanced search capabilities, aligning with the theoretical conclusions discussed in the main text.
29

---

Table 11: Results of Random Data Forgetting(30%) for various methods of unlearning Resnet 18 for Cifar 100 classification. The results are the mean value over 10 independent trials. The best performance highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
29.00
99.97
71.47
52.22
0,00
FT
30.27 (1.27)
82.62 (17.35)
61.14 (10.33)
31.93 (20.29)
12.31
RL
26.14 (2.86)
98.12 (1.85)
63.46 (8.01)
66.68 (14.46)
6.79
GA
3.07 (25.93)
97.45
75.24 (3.77)
6.90 (45.32)
19.39
IU
10.82 (18.18)
90.43
65.30 (6.17)
17.50 (34.72)
17.15
BE
3.00 (26.00)
97.24 (2.73)
73.68 (2.21)
9.64 (42.58)
18.38
BS
2.95 (26.05)
97.32 (2.65)
73.81 (2.34)
8.84 (43.38)
18.61
spasee
2.57 (26.43)
97.96 (2.01)
75.96 (4.49)
6.17 (46.05)
19.75
SalUn
24.08 (4.92)
98.19 (1.78)
61.74 (9.73)
65.16 (12.94)
7.34
SalUn-soft
25.39 (3.61)
97.59 (2.38)
62.12 (9.35)
62.49 (10.27)
6.40
GDRGMA
27.04 (1.96)
98.37 (1.60)
60.90 (10.57)
70.29 (18.07)
8.05
EUPMU
17.68 (11.32)
98.96 (1.01)
08.06
50,93 (1.29)
4.16
EUPMU-fast
19.64 (9.36)
97.81 (2.16)
62.53 (8.94)
56.06 (3.84)
6.08
Table 12: Results of Random Data Forgetting(50%) for various methods of unlearning Resnet 18 for Cifar 100 classification. The results are the mean value over 10 independent trials. The best performance is highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
33.28
99.98
67.75
61.21
0.00
FT
20.84 (12.44)
91.03 (8.95)
(3.71)
30.20 (31.01)
14.03
RL
22.48 (10.80)
95.51 (4.47)
54.52 (13.23)
53.41 (7.80)
9.07
GA
2.59 (30.69)
97.53 (2.45)
75.38 (7.63)
6.07 (55.14)
23.98
IU
6.76 (26.52)
93.58 (6.40)
67.80 (0.05)
13.06 (48.15)
20.28
BE
5.92 (27.36)
24.28
67.00 (0.75)
17.13 (44.08)
19.47
BS
6.22 (27.06)
94.54 (5.44)
66.62((113)
16.57 (44.64)
19.57
sp-spe
2.48 (30.80)
(8.76(1.22)
15.94(8.19)
6.59 (54.62)
23.71
SalUn
28.56 (4.72)
96.31 (3.67)
52.88 (14.87)
62.88 (1.67)
6.23
SalUn-soft
24.84 (8.44)
95.44 (4.54)
54.32 (13.43)
55.50 (5.71)
8.03
GDRGMA
22.66 (10.62)
95.48 (4.50)
53,34 (14.41)
53.33 (7.88)
9.35
EUPMU
21.57 (11.71)
98.97 (1.01)
64.13 (3.62)
58.43 (2.78)
4.78
EUPMU-fast
18.68 (14.60)
94.20 (5.78)
54.29 (13.46)
40.91 (20.30)
13.54
Limitations
While proposed method demonstrates significant improvements over existing unlearning ap- proaches, two inherent constraints persist: 1) Theoretically, we aim to achieve Pareto-optimal solutions that balance unlearning and utility objectives through error tolerance (ex) calibration. How- ever, the absence of explicit patterns in these dual objectives prevents the formal derivation of their interrelationship within theoretical frameworks. Rigorous mathematical formulation would require introducing domain -spacfcc assumptions about objective correlations, which inherently narrows the theoretical applicability. After careful consideration, we maintain the original formulation to preserve methodological generality; 2) Practically, the non-zero directly governs the trade-off between utility preservation and unlearning completeness. As shown in our ablation studies (Figure 6 this hyperparameter critically determines empirical performance. Nevertheless, lacking theoretical guidance for optimal ét, practical implementation relies on empirical tuning through grid search Bayesian optimization. This dual constraint-hheortiiall indeterminacy and empirical dependency-dedt the current practical boundary of our framework.
tolerance varepsilon, 0
G Broader Impacts
This paper addresses the core unlearning-utiiie tradeoff in machine unlearning, aiming to enhance its practical deployability while effectively mitigating challenges posed by generative models producing
30

---

Table 13: Results of Random Data Forgetting( 10%) for various methods of unlearning Resnet for classification. The results are the mean value over 10 independent trials. The best performance highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
36.54
99.98
63.71
64.57
0.00
FT
6.89 (29.65)
98.80 (1.18)
65.05 (1.34)
16.11 (48.46)
20.16
RL
28.40 (8.14)
98.35 (1.63)
61.17 (2.54)
57.80 (6.77)
4.77
GA
5.42 (31.12)
95.10 (4.88)
65.91 (2.20)
14.46 (50.11)
22.08
IU
11.79 (24.75)
89.83 (10.15)
61.37 (2.34)
14.98 (49.59)
21.71
BE
6.32 (30.22)
93.92 (6.06)
63.99 (0.28)
15.33 (49.24)
21.45
BS
6.28 (30.26)
93.96
64.15 (0.44)
15.50 (49.07)
21.45
spsspae
5,91 (30.63)
95.10 (4.88)
(2.44)
13.96 (50.61)
22.14
SalUn
33.43 (3.11)
97.61 (2.37)
61.21 (2.50)
56.27 (8.30)
4.07
SafUn-soft
32.25 (4.29)
91.01 (8.97)
58.65 (5.06)
33.60 (30.97)
12.32
GDRGMA
19.59 (16.95)
97.91 (2.07)
61.91 (1.80)
45.10 (19.47)
10.07
EUPMU
35.18 (1.36)
97.62 (2.36)
60.51 (3.20)
(8.47)
3.85
EUPMU-fast
30.18 (6.36)
97.12 (2.86)
60.81 (2.90)
49.37 (15.20)
6.83
Table 14: Results of Random Data Forgetting(30%) for various methods of unlearning Resnet 18 for Tiny- Imagenttt classification. The results are the mean value over 10 independent trials. The best performance is highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
39.00
99.98
60.81
67.30
0.00
FT
9.75 (29.25)
99.05 (0.93)
64.41 (3.60)
25.65 (41.65)
18.86
RL
13.11 (25.89)
90.32 (9.66)
56.63 (4.18)
33.33 (33.97)
18.43
GA
5.14 (33.86)
95.08 (4.90)
66.01 (5.20)
14.38 (52.92)
24.22
IU
6.96 (32.04)
93.27 (6.71)
63.75 (2.94)
13.90 (53.40)
23.77
BE
10.84 (28.16)
89.12
59.79 (1.02)
18.82 (48.48)
22.13
BS
8.85 (30.15)
91.06 (8.92)
62.84 (2.03)
17.74 (49.56)
22.66
s-ssprse
5.15 (33.85)
97.31 (2.67)
(5.10)
13.61 (53.69)
23.83
SalUn
33.60 (5.40)
96.05 (3.93)
54.57 (6.24)
46.77 (20.53)
9.03
SalUn-soft
33.99 (5.01)
87.75 (12.23)
52.77 (8.04)
37.13 (30.17)
13.86
GDRGMA
22.52 (16.48)
96.00 (3.98)
55.27 (5.54)
38.96 (28.34)
13.59
EUPMO
40.50 (1.50)
94.05 (5.93)
50.71 (10.10)
50.61 (16.69)
8.55
EUPMU-fast
33.92 (5.08)
89.48 (10.50)
53.21 (7.60)
46.23 (21.07)
11.06
infringing or non-compliant content. By optimizing the balance between knowledge retention and targeted erasure, our methodology directly tackles the dual imperatives of model governance and regulatory compliance. The technical focus on improving governance capabilities inherently minimizes adverse societal impacts, as the proposed approach does not introduce systemic biases or operational disruptions. This dual focus not only broadens the practical applicability of machine unlearning across industries but also establishes framework for ethically constrained Al development, aligning with regulatory demands for content accountability without compromising model utility.
31

---

Table 15: Results of Random Data Forgetting(50%) for various methods of unlearning Resnet for Imay-em classification. The results are the mean value over 10 independent trials. The best performance highlighted in bold.
Methods
UA
RA
TA
MIA
Avg. Gap
Retrain
43.15
99.99
56.29
71.15
0.00
FT
7.13 (36.02)
99.32 (0.67)
65.27 (8.98)
18.09 (53.06)
24.68
RL
24.30 (18.85)
85.00 (14.99)
48.47 (7.82)
25.75 (45.40)
21.77
GA
4.63 (38.52)
95.35 (4.64)
66.29 (10.00)
12.96 (58.19)
27.84
IU
18.00 (25.15)
83.21 (16.78)
56.41 (0.12)
20.57 (50.58)
23.16
BE
7.56 (35.59)
92.47 (7.52)
62.77 (6.48)
16.62 (54.53)
26.03
BS
8.87 (34.28)
90.68 (9.31)
59.03 (2.74)
20.21 (50.94)
24.32
spspaes
4.44 (38.71)
95.76 (4.23)
66.39 (10.10)
12.76 (58.39)
27.86
SalUn
44.84 (1.69)
94.29 (5.70)
45.83 (10.46)
54.73 (16.42)
8.57
SalUn-soft
33.56 (9.59)
88.25 (11.74)
48.15 (8.14)
39.76 (31.39)
15.21
GDRGMA
28.73 (14.42)
91.16 (8.83)
47.73 (8.56)
36.28 (34.87)
16.67
EUPMU
31.97 (11.18)
(1.49)
52.71 (3,58)
$3.18
8.55
EUPMU-fast
33.60 (9.55)
88.59 (11.40)
49.71 (6.58)
39.86 (31.29)
14.71
Methoos
Porgetängclass: Airplane
C1
C2
C3
C4
C5
C6
C7
CB
C9
Random
Salun
Ours
Figure Image generations unlearning for DDPM on CIFAR-10. The forgetting class is given by 'airplane', and 'C" refers to the non-forgetting class name, e.g., 'car' (CI).
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction in this paper accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limitation discussion in Appendix F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification: We provide the full set of assumptions and complete (and correct) proof in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the experimental information in Section
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necesssary to either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

reproducible, they Answer: [Yes]
Justification:
We provide our code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides http://ooaeaaiioiiinnnneeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips httpcc/cpplllgggiiieee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments are averaged from multiple experiments, and the std is very small, so there no need report the error bar.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
35

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We note that all experiments are carried out on two A100 GPUs in Section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwww....oooo///ccchccteea
Answer: [Yes]
Justification:
This paper focuses on thcoretical results and does not involve ethical issues.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal
impacts in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
38

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, non- standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
39


Reviews:
[{"review_id": "RLSOC7Jvlg", "reviewer": "Reviewer_RLwG", "summary": "This paper aims to addresses the challenge of utility-preserving in machine unlearning. It first identifies the limitations of the conventional linearization approach, which suffers from utility loss, and the multi-objective optimization approach, which leads to insufficient unlearning and efficiency issues. To overcome these, the authors propose an Implicit Gradient Surgery method. This method features utility-preserving characteristics and guarantees sufficient optimization of the forgetting objective. Furthermore, an approximation technique within the method ensures computational efficiency comparable to the linear weighting approach. The paper also provides a Pareto convergence analysis, proving that the proposed algorithm achieves the aforementioned design goals. The experimental results robustly validate this theoretical finding.", "questions": "1. While the paper claims computational efficiency comparable to the linearization method, Table 5 shows that EUPMU still has a slightly higher RTE than Linearization. What causes this discrepancy?\n2. The proposed method appears broadly applicable to all machine unlearning scenarios, yet the experiments only demonstrate results on image classification and generation tasks. What is its applicability in language models?\n3. If error tolerance is disabled, would the results be comparable to those of the multi-objective optimization method?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The problem formulation and motivation are presented with clarity. Figure 1 illustrates the limitations of both linearization and multi-objective optimization methods in machine unlearning scenarios. The background section provides a detailed analysis, offering valuable insights into addressing the core challenge of utility-preserving. This convincingly motivates the necessity of the proposed EUPMU framework.\n2. This paper proposes an interesting and intuitive methodological framework. Starting from the constrained optimization formulation in Section 3, the derivation seamlessly transitions to the design of Explicit Unilateral Gradient Surgery, making its geometric interpretation highly accessible. The approximation technique, based on solving a dual problem, cleverly avoids the need for separate gradient computations for multiple objectives.\n3. The authors provide rigorous theoretical analysis for proposed method, such as the proofs for both Pareto optimality (convex case) and Pareto stationarity (non-convex case). Remark 3.6 further deduces the algorithm’s properties from these results, demonstrating that it achieves sufficient optimization of the forgetting objective while maintaining predetermined utility loss bounds. This theoretical guarantee directly resolves the core problem identified in the study.\n4. The comprehensive experimental results robustly support the theoretical claims. The paper provides extensive tests on Pareto optimality, concept removal efficacy, and computational efficiency, offering empirical evidence for the algorithm’s superiority across all key metrics.\n\n**Weaknesses:**\n1. The introduction of two new hyperparameters in EUPMU increases the complexity of practical tuning.\n2. The proposed Explicit Unilateral Gradient Surgery bears resemblance to prior techniques, which diminishes its perceived originality compared to novel contributions in the field.", "was_revised": true, "final_justification": "The authors' response provides many valuable insights, which have further enhanced my understanding of the paper's novelty and underlying principles. Therefore, I decide to raise my score.", "rebuttal": "We would like to thank Reviewer RLwG for acknowledging our novelty and contributions. We address the concerns as following:\n\n**W1: The introduction of two new hyperparameters in EUPMU increases the complexity of practical tuning.** \n\nAlthough our proposed EUPMU method introduces two hyperparameters, $\\varepsilon_t$ and $\\beta_t$​, these are essential for addressing two core challenges in machine unlearning.\n\nOn one hand, machine unlearning faces the challenge of utility degradation. While a multi-objective approach can prevent utility degradation, it often fails to ensure sufficient forgetting of target information. Therefore, we introduce $\\varepsilon_t$​ to further balance utility preservation and unlearning effectiveness.\n\nOn the other hand, introducing a multi-objective approach typically incurs an additional round of backpropagation. To mitigate this, we adopt a gradient-based approximation that avoids this extra backward pass, for which we introduce $\\varepsilon_t$ as the gradient step size in the approximation.\n\nMoreover, both hyperparameters are guided by theoretical insights. The tuning of $\\varepsilon_t$​ can refer to Remark 3.3 and Theorem 3.7, which provide theoretical foundations for determining its value based on the scale of the loss function. Theorem 3.4 also offers a feasible range for $\\beta_t$​ to ensure the effectiveness of the approximation, and empirical testing shows this range works well in practice.\n\nIn summary, these two new hyperparameters are necessary and, thanks to theoretical guidance, can be tuned within a relatively small and manageable range, thereby avoiding excessive tuning overhead.\n\n**W2: The proposed Explicit Unilateral Gradient Surgery bears resemblance to prior techniques, which diminishes its perceived originality compared to novel contributions in the field.** \n\nWhile Explicit Unilateral Gradient Surgery is related to methods in \\[1,2], our approach differs in two fundamental ways: formulation and utility control.\n\n* Fundamental Difference (Principled Formulation vs. Heuristic Fixes):\nPrior methods, such as \\[1,2], stem from a multi-task learning (MTL) perspective, identify issues like gradient conflict, and propose heuristic fixes. GDR-GMA \\[1] uses separate handcrafted modules for rectification (GDR) and weighting (GMA). Learn to Unlearn \\[2] projects gradients away from a pre-computed Core Gradient Space (CGS). These are fundamentally gradient correction methods. In contrast, our method does not begin with gradient operations. We formulate a new constrained optimization problem: the Utility-Preserving Unlearning Problem (UPUP). Our Unilateral Gradient Surgery is not a heuristic tweak but the exact analytical solution that emerges from solving the UPUP problem. This principled formulation is the core of our contribution.\n\n* Utility Control (Proactive Constraint vs. Reactive Balancing):\nOur method includes an explicit constraint $\\varepsilon$, a user-defined utility loss budget derived from UPUP, offering an interpretable and proactive control mechanism. In comparison, other approaches rely on reactive heuristic modules like GMA, which adjust based on observed loss changes but lack pre-specifiable utility-preservation levels. Overall, our method shifts the paradigm from reactive balancing to proactive constrained optimization—crucial for dependable and predictable deployments.\n\n**Q1: While the paper claims computational efficiency comparable to the linearization method, Table 5 shows that EUPMU still has a slightly higher RTE than Linearization. What causes this discrepancy?**\n\nGreat question! This slight discrepancy arises because, although we avoid explicitly solving both objectives by using a gradient approximation to the Lagrangian equation (Equation 4), the computation of \\$\\ell\\_r(\\theta\\_{t+1})\\$ (Equation 8) introduces one additional forward pass. This is why the runtime is slightly higher than the linearization method—but still much more efficient than backward passes.\n\nFurthermore, we can eliminate this forward pass by reusing \\$\\ell\\_r(\\theta\\_{t+1})\\$ from the next time step. Although the data batches at times $t$ and $t+1$ may differ due to randomness, the stochastic nature is unbiased and does not affect convergence. We plan to elaborate and test this in future versions.\n\n**Q2: The proposed method appears broadly applicable to all machine unlearning scenarios, yet the experiments only demonstrate results on image classification and generation tasks. What is its applicability in language models?** \n\nExcellent observation! In fact, our method proposes a general utility-preservation mechanism that applies to most machine unlearning settings, including LLMs. We used vision models in this version to validate effectiveness comprehensively. We plan to include experiments on LLMs in future versions.\n\n\n**Q3: If error tolerance is disabled, would the results be comparable to those of the multi-objective optimization method?** \n\nSetting $\\varepsilon$ = 0 reduces UPUP to strict utility preservation. We provide the experiments with different error tolerance below.\n\n| $\\varepsilon$ | Retain Loss | Forget Loss |\n| :---: | :---: | :---: |\n| 0.00  | 0.052  |   **2.25** | \n| 0.10 | 0.035  | 2.53  | \n| 0.25 | 0.031  | 2.92  | \n| 0.50 | 0.029  | 3.58  | \n| 1.00 | **0.028** | 4.41  | \n\nFrom the above results, we know that if error tolerance is disabled, the Retain Loss increases significantly, which indicates that the utility has been degraded seriously. Therefore, it would be better to choose an appropriate error tolerance in a suitable interval as introduced in the answer of W1.\n\n[1]Lin, Shen, et al. \"GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2] Qu, Youyang, et al. \"Learn to Unlearn: A Survey on Machine Unlearning.\" arXiv preprint arXiv:2305.07512 (2023)."}, {"review_id": "uoNBdFqi7P", "reviewer": "Reviewer_DBBQ", "summary": "This paper addresses the Utility-Preserving Unlearning Problem by formulating unlearning as a constrained optimization task. It proposes Unilateral Gradient Surgery to balance unlearning and utility preservation, and further introduces an efficient implicit variant to reduce computational cost. The authors provide theoretical analysis proving convergence to Pareto optimality, and experiments validate the method’s effectiveness.", "questions": "1.\tFurther clarifying the contributions of this paper, particularly in comparison to existing gradient conflict mitigation approaches. The current explanation that the difference lies in the principle origin is not entirely convincing. It would be helpful to elaborate on this distinction more clearly, ideally within the main body of the paper, as this would better highlight the novelty of the proposed method for the reader.\n\n2.\tThe issues related to the evaluation metrics should be revisited and addressed appropriately.\n\n3.\tIt would be helpful to further enhance the experimental section by providing additional results and including ablation studies on key hyperparameters.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.The paper is well-written and well-structured, with helpful illustrations that make the core ideas more accessible.\n\n2.The paper provides thorough theoretical analysis to support the proposed solution, which strengthens the soundness and credibility of the approach.\n\n3.The paper presents several practical unlearning scenario settings, which enhance the applicability of the proposed method and provide empirical evidence of its effectiveness.\n\nWeaknesses:\n\n1.The idea of leveraging gradient conflict mitigation for unlearning has been explored in prior work, which somewhat limits the novelty of this paper. Although the authors argue that their approach is motivated by a different perspective, I think this justification not entirely convincing. Additionally, the paper lacks discussion of several closely related works, such as GDR-GMA[1] and G-effect[2]. Notably, GDR-GMA also considers the unlearning process as a multi-task optimization problem.\n\n2.The paper lacks ablation studies on key hyperparameters, such as the error tolerance. Although the authors mention in the limitations section that related results are shown in Figure 7, I could not find any description in Figure 7 pertaining to error tolerance.\n\n3.The evaluation metrics used in this paper, at least for the classification tasks, raise some concerns and may not fully align with the goals of unlearning, which affects the overall assessment of the work. Specifically, treating lower UA and MIA scores and higher RA and TA scores as universally better may be misleading. Under such criteria, retraining might appear to perform poorly as an unlearning method. Moreover, overly aggressive unlearning might introduce new attack surfaces—for example, by exposing abnormal behavior on forgotten data—which highlights the necessity of maintaining a certain level of generalization. A rigorous unlearning approach should aim to closely match the behavior of retraining, rather than simply minimizing or maximizing certain metrics. \n\n[1]Lin, Shen, et al. \"GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2]Wang, Qizhou, et al. \"Rethinking llm unlearning objectives: A gradient perspective and go beyond.\" arXiv preprint arXiv:2502.19301 (2025).", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer DBBQ for acknowledging our novelty and contributions. We address the concerns as following:\n\n**W1 \\& Q1: The idea of leveraging gradient conflict mitigation for unlearning has been explored in prior work, which somewhat limits the novelty of this paper**\n\nIn fact, the starting point of our method is not gradient conflict mitigation. Instead, we approach the problem from the perspective of how to fully optimize the unlearning process under the constraint of *utility preservation*. We start by formulating a novel constrained optimization problem: the Utility-Preserving Unlearning Problem (UPUP).\n\nAlthough the resulting gradient update rule bears resemblance to methods like gradient surgery, our starting point is more fundamental. This foundational approach allows us to derive theoretical insights. For instance, *Remark 3.3* provides an estimate of the extent to which utility is degraded during the unlearning process. Furthermore, *Remark 3.6*, from the perspective of Pareto Optimality, demonstrates that the unlearning objective can be fully optimized under the utility constraint.\n\nAdditionally, we believe that our starting point and analytical framework can provide more fundamental research avenues for this field. For example, while our current solution to the UPUP problem (Equation 2) employs a first-order Taylor expansion, our formulation naturally extends to *second-order expansions*. This could lead to the development of second-order or quasi-second-order unlearning algorithms. As another example, from a regularization standpoint, one could apply customized constraints to the update direction $d_t$, such as incorporating *user preferences* into the UPUP formulation, which would also lead to novel algorithms.\n\nWe thank Reviewer DBBQ for pointing out some missing related works:\n*   Compared to GDR-GMA[1], this method stems from a multi-task learning (MTL) perspective, identifies issues like gradient conflict, and proposes heuristic fixes. While EUPMU is derived from the proposed UPUP optimization problem. Additionally, GDR-GMA \\[1] uses separate handcrafted modules for rectification (GDR) and weighting (GMA). Hence, GDR‑GMA needs two backward passes, adding approximately 30% more compute time with the same amount of data as EUPMU, which requires only one backward pass.\n*   Compared to G-effect [2], their \"G-effect\" toolkit provides a powerful gradient-based framework for understanding unlearning. The primary distinction lies in the objective: G-effect is a novel analytical toolkit designed to diagnose and understand the impacts of existing unlearning objectives. In contrast, our EUPMU is a novel constructive algorithm derived from a first-principles constrained optimization problem (UPUP). Our work provides an explicit, controllable mechanism ($\\varepsilon$) to navigate the utility-unlearning trade-off, which is a constructive solution motivated by the kind of insights G-effect aims to provide. We will add a detailed discussion of this complementary work to our related works section.\n\nTherefore, we argue that the principle we propose is fundamentally different from existing gradient conflict mitigation methods. Rather than being a surface-level application of gradient projection, our foundational framework is more capable of inspiring and guiding future research. We will also add more discussion on recent related works in the new version.\n\n**W2 \\& Q3: The paper lacks ablation studies on key hyperparameters, such as the error tolerance. Although the authors mention in the limitations section that related results are shown in Figure 7, I could not find any description in Figure 7 pertaining to error tolerance.**\n\nTo address the concern about the lack of ablation studies, we have conducted experiments on the key hyperparameters, including the error tolerance $\\varepsilon$. We present the results below and will add a detailed discussion to the appendix of our paper.\n\n* Sensitivity to the error tolerance $\\varepsilon$\n\nWe analyzed the impact of the error tolerance $\\varepsilon$, which controls the trade-off between model utility and unlearning effectiveness. The results are shown in the table below:\n\n| $\\varepsilon$ | Retain Loss | Forget Loss |\n| :---: | :---: | :---: |\n| 0.00  | 0.052  |   **2.25** | \n| 0.10 | 0.035  | 2.53  | \n| 0.25 | 0.031  | 2.92  | \n| 0.50 | 0.029  | 3.58  | \n| 1.00 | **0.028** | 4.41  | \n\nAs shown, increasing $\\varepsilon$ leads to a lower **Retain Loss** (better utility) at the cost of a higher **Forget Loss** (less effective unlearning). This demonstrates that $\\varepsilon$ serves as a controllable lever to navigate the Pareto front of the utility-unlearning trade-off. This allows practitioners to tune the algorithm based on specific downstream requirements, which is consistent with our theoretical analysis in Remark 3.3 and Remark 3.6.\n\n* Sensitivity to the step-size $\\beta$\n\nWe also investigated the sensitivity to the step-size $\\beta$, which is used in the gradient approximation for solving $\\lambda$.\n\n|  $\\beta$ | Retain Loss | Forget Loss  |\n| :---: | :---: | :---: |\n|   0.01 | 0.043  |   **2.41** |\n|   0.05 | 0.034  | 2.68  |\n|   0.10 | 0.032  | 2.80  |\n|   0.50 | 0.031  | 2.92  | \n|   1.00 | **0.030** | 2.92  | \n\nThe results indicate that our method is robust to the choice of $\\beta$. The overall performance shows minimal variation across a wide range of values. This empirical finding aligns with our convergence analysis in Theorem 3.4, which proves that the algorithm converges to a Pareto-optimal solution as long as $\\beta$ is chosen within a theoretically sound range.\n\nFurthermore, the selection of these hyperparameters is not arbitrary but is guided by our theory. The search range for $\\varepsilon$ can be informed by Remark 3.3 and Theorem 3.7, while Theorem 3.4 provides guidance for selecting a suitable $\\beta$ to ensure convergence. We will add these results and a comprehensive discussion to the revised paper. Thank you for helping us improve the clarity and completeness of our work.\n\n**W3 \\& Q2: The evaluation metrics for the classification tasks may not fully align with the goals of unlearning.**\n\nWe thank the reviewer for this insightful comment. We agree that relying solely on individual classification metrics can be insufficient, as the core challenge of unlearning lies in the trade-off between forgetting efficacy and retaining utility.\n\nTo align with the goals of unlearning, the community often measures a method's performance against a \"Retrain Oracle\"—a model retrained from scratch on only the retain data.\n\nHowever, we argue that the gap-to-retrain metric could be fundamentally flawed in its practical application. In reality, most methods do not have any predictive capability about what the retrained model will look like. What ends up happening is a kind of “target painting after the shot”—researchers run the retrained model first, then retroactively select a run from their own method that happens to be close to it.\n\nTherefore, in our work, **we tuned our hyperparameters strictly based on achieving the best practical performance**—maximizing Retain/Test Accuracy while minimizing Forget Accuracy and MIA score—without any knowledge of the Retrain Oracle.\n\nHere’s the table with the *gap to Retrain Oracle* shown in parentheses for every metric, plus an *Avg Gap* column (mean of the four gaps per method), this is class-wise unlearning for Resnet18 on Cifar10:\n\n| Method | Retain Acc | Forget Acc | Test Acc | SVC MIA  | Avg Gap  |\n| --- | --- | --- | --- | --- | --- |\n| Retrain Oracle | 99.42 % (0.00) | 0.00 % (0.00) | 94.08 % (0.00) | 0.000 (0.000) |   0.00 |\n| GDR-GMA  | 95.30 % (4.12) | 0.00 % (**0.00**) | 88.70 % (5.38) | 0.000 (0.000) |   2.38 |\n| Learn to Unlearn | 98.98 % (**0.44**) | 0.76 % (0.76) | 92.00 % (2.08) | 0.000 (0.000) |   0.82 |\n| **EUPMU(ours)** | 98.75 % (0.67) | 0.09 % (0.09) | 92.18 % (**1.90**) | 0.000 (0.000) |   **0.67** |\n\nWe also tested Resnet 18 unlearning on Cifar 100, random data forgetting 10%.\n| **Methods** | **UA** | **RA** | **TA** | **MIA** | **Avg. Gap 4** | **Avg. Gap 2** |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Retain | 26.47 | 99.97 | 74.13 | 51.00 | 0.00 | 0.00 |\n| FT | 2.42 (24.05) | 99.95 (**0.02**) | 75.55 (1.42) | 11.04 (39.96) | 16.36 | 20.69 |\n| RL| 55.03 (28.56) | 99.81 (0.16) | 70.03 (4.09) | 98.97 (47.97) | 20.20 | 26.03 |\n| GA | 3.13 (23.34) | 97.33 (2.64) | 75.31 (1.18) | 7.24 (43.76) | 17.73 | 22.47 |\n| IU | 3.18 (23.29) | 97.15 (2.82) | 73.49 (0.64) | 9.62 (41.38) | 17.03 | 21.01 |\n| BE | 2.31 (24.16) | 97.27 (2.70) | 73.93 (**0.20**) | 9.62 (41.38) | 17.11 | 20.79 |\n| BS | 2.27 (24.20) | 97.41 (2.56) | 75.26 (1.13) | 5.82 (45.18) | 18.27 | 23.16 |\n| l₁-Sparse | 10.64 (15.83) | 96.62 (3.35) | 70.99 (3.14) | 22.58 (28.42) | 12.68 | 15.78 |\n| SalUn | 27.53 (**1.06**) | 97.00 (2.97) | 67.79 (6.34) | 70.79 (19.79) | 7.54 | 13.07 |\n| SalUn-soft| 24.24 (2.23) | 98.95 (1.02) | 70.48 (3.65) | 79.13 (28.13) | 8.76 | 15.89 |\n| **EUPMU(ours)** | 13.53 (12.94) | 97.02 (2.95) | 71.90 (2.23) | 51.02 (**0.02**) | **4.53** | **1.12** |\n\n\nDespite this more rigorous and realistic tuning methodology, our method proves to be the most aligned with the theoretical ideal.\n\nIf we are going to tune the model to minimize the gap to retrain, we can also search for the best $\\varepsilon$ value to balance the unlearning benchmark to a desired value while keeping the retaining performance as close as possible to the retrain.\n\n\n\n[1]Lin, Shen, et al. \"GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2]Wang, Qizhou, et al. \"Rethinking llm unlearning objectives: A gradient perspective and go beyond.\" arXiv preprint arXiv:2502.19301 (2025)."}, {"review_id": "X74rKecRSr", "reviewer": "Reviewer_mcr2", "summary": "This paper addresses the challenge of balancing unlearning efficacy and utility preservation in machine unlearning (MU). The authors formulate the Utility Preserving Unlearning Problem as a constrained optimization: maximize the decrease in the unlearning loss under a bounded increase in the retaining loss. Empirical evaluations on image classification and generation tasks demonstrate that EUPMU outperforms baselines.", "questions": "Please see the weaknesses.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. It is insightful that formulating MU as a constrained optimization problem with explicit control over utility loss.\n2. This paper provides detailed theoretical analysis and experimental evaluations that show consistent performance improvements.\n\nWeakness:\n\n1. Although it is very interesting to formulate MU as a gradient surgery problem, the gradient projection methods on MU have been investigated in the existing works [1][2]. I suggest that the authors add the corresponding discussions and comparison experiments.\n2. Some important methods are missed on the SD models, e.g., MACE [3] and UCE [4].\n3. The ablation experiments on the hyperparameters should also be added.\n4. Is it possible to implement EUPMU for multi-concept unlearning on the SD models?\n\n[1] Lin S, Zhang X, Susilo W, et al. GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 9087-9095.\n\n[2] Hoang T, Rana S, Gupta S, et al. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 4819-4828.\n\n[3] Lu S, Wang Z, Li L, et al. Mace: Mass concept erasure in diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6430-6440.\n\n[4] Gandikota R, Orgad H, Belinkov Y, et al. Unified concept editing in diffusion models[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 5111-5120.", "was_revised": true, "final_justification": "The authors' responses have addressed my concerns, and thus I decide to maintain my score.", "rebuttal": "We would like to thank Reviewer mcr2 for the acknowledgement and insightful comments. We address the concerns as follows.\n\n**W1. Gradient projection methods already exist...add the corresponding discussions and comparison experiments.**\n\nContributive question! We now explicitly compare EUPMU with GDR‑GMA[1] and Learn to Unlearn. The key differences are:\n\n* Motivation:  Previous works follow the idea gradient conflict mitigation, while EUPMU is the analytic solution of the constrained optimisation problem (UPUP, Equation 2). By re-framing MU at this more fundamental optimization level, we open up new avenues for developing more advanced algorithms. Future work could explore second-order Taylor expansions or introduce preference vectors to solve the UPUP problem. Such approaches can lead to more efficient and practical algorithms, a potential not provided by the singular focus on gradient conflict.\n\n* Theoretical Insights: While the final update mechanism is analogous to methods like gradient surgery, our approach originates from a more principled formulation. This foundational basis enables us to establish formal theoretical guarantees. Specifically, *Remark 3.3* quantifies the potential degradation in model utility during unlearning. Moreover, *Remark 3.6* leverages the concept of Pareto Optimality to prove that the unlearning objective can be fully satisfied without violating the utility constraint.\n\n\n* Efficiency:  GDR‑GMA needs two, and Learn‑to‑Unlearn adds an SVD pre‑pass, while EUPMU requires a single backward pass.  On ResNet‑18/CIFAR‑10 EUPMU is ≈ 30 % faster than GDR‑GMA[1] and ≈ 1.9 times faster than Learn‑to‑Unlearn[2] while achieving 0.09 % UA vs 0.76 %.\n\nHere’s the results of class-wise unlearning for Resnet18 on Cifar10:\n\n| Method | Retain Acc | Forget Acc | Test Acc | MIA  | Avg Gap  |\n| --- | --- | --- | --- | --- | --- |\n| Retrain Oracle | 99.42 % (0.00) | 0.00 % (0.00) | 94.08 % (0.00) | 0.000 (0.000) |   0.00 |\n| GDR-GMA  | 95.30 % (4.12) | 0.00 % (**0.00**) | 88.70 % (5.38) | 0.000 (0.000) |   2.38 |\n| Learn to Unlearn | 98.98 % (**0.44**) | 0.76 % (0.76) | 92.00 % (2.08) | 0.000 (0.000) |   0.82 |\n| **EUPMU (Ours)** | 98.75 % (0.67) | 0.09 % (0.09) | 92.18 % (**1.90**) | 0.000 (0.000) |   **0.67** |\n\nBased on these results, our method demonstrates a superior trade-off between model utility and unlearning effectiveness. Furthermore, it achieves Pareto superiority over the other methods, which is consistent with our theoretical analysis. We will incorporate this analysis and these experimental results into the revised manuscript.\n\n**W2. MACE and UCE are missing on Stable Diffusion**\n\nGood suggestion! We have added SD experiments, using the code base of MACE[4] to implement EUPMU on unlearning the 10 celebrities' faces, as shown in the following results.\n\n| Method | Retain Accuracy | Forget Accuracy | Harmonic Mean |\n|---|---|---|---|\n| UCE | 0.8448 | **0.0000** | 0.9159 |\n| MACE | 0.9359 | 0.0040 | 0.9650 |\n| **EUPMU (Ours)** | **0.9375** | 0.0040 | **0.9659** |\n\nThe results above indicate that EUPMU, as a more advanced optimization mechanism, achieves better performance retention than MACE and strikes a superior utility-unlearning trade-off compared to UCE[5]. We will provide more comprehensive experiments in the revised version of our paper.\n\n**W3. Ablate the key hyper‑parameters**\n\nBelow we provide an expanded ablation study  the two hyperparameters of EUPMU as follows.\n\n* Sensitivity to the error tolerance $\\varepsilon$\n\n| $\\varepsilon$ | Retain Loss | Forget Loss |\n| :---: | :---: | :---: |\n| 0.00  | 0.052  |   **2.25** | \n| 0.10 | 0.035  | 2.53  | \n| 0.25 | 0.031  | 2.92  | \n| 0.50 | 0.029  | 3.58  | \n| 1.00 | **0.028** | 4.41  | \n\nThe results highlight the inherent trade-off between utility and unlearning, which is directly governed by $\\varepsilon$. Specifically, a higher $\\varepsilon$ prioritizes utility preservation at the expense of unlearning performance. This confirms that $\\varepsilon$ functions as a practical control knob, enabling users to balance these competing objectives according to their requirements—a finding that reflects our theoretical analysis in Remarks 3.3 and 3.6.\n\n* Sensitivity to the step-size $\\beta$\n\n|  $\\beta$ | Retain Loss | Forget Loss  |\n| :---: | :---: | :---: |\n|   0.01 | 0.043  |   **2.41** |\n|   0.05 | 0.034  | 2.68  |\n|   0.10 | 0.032  | 2.80  |\n|   0.50 | 0.031  | 2.92  |\n|   1.00 | **0.030** | 2.92  | \n\nOur method exhibits strong robustness to the choice of $\\beta$. We observed negligible variation in overall performance even when $\\beta$ was varied significantly. This experimental result aligns perfectly with the convergence guarantee in Theorem 3.4, which proves that a Pareto-optimal solution is attainable for any $\\beta$ selected from a theoretically valid range.\n\nFurthermore, the selection of these hyperparameters is not an arbitrary process but is instead principled and guided by our theoretical framework. For instance, the search range for $\\varepsilon$ can be effectively constrained by the insights from Remark 3.3 and Theorem 3.7. Similarly, Theorem 3.4 provides a clear basis for selecting a suitable $\\beta$ that ensures convergence. We will incorporate these results and a comprehensive discussion into the revised manuscript. We thank you for this valuable suggestion, which has helped us improve the clarity and completeness of our work.\n\n**W4. Can EUPMU erase multiple concepts on SD?**\n\nInsightful Question! Yes. There are two primary approaches to extend our method to multi-concept unlearning:\n\n* Sequential Unlearning: The first approach is to erase multiple concepts sequentially, one after another. In fact, Figure 4 and Table 3 of our paper already demonstrate successful unlearning for two and three concepts using this method.\n\n* Simultaneous Unlearning: The second approach is to erase all concepts in a single step. This requires extending the UPUP problem into a constrained multi-objective optimization problem. For example, for two-concept removal, the problem is formulated as follows:\n\n$$\\max_{\\boldsymbol{d}_t} \\frac{1}{\\alpha_t} \\min [  r_u^1 (\\alpha_t, \\boldsymbol{d}_t) , r_u^2 (\\alpha_t, \\boldsymbol{d}_t) ] - \\frac{1}{2} \\left\\|\\|\\boldsymbol{d}_t\\|\\right\\|^2 \n\\text{ s.t. }\\frac{1}{\\alpha_t}  r_r(\\alpha_t, \\boldsymbol{d}_t) \\geq - \\varepsilon_t$$\n\nThe corresponding first-order Taylor expansion derived from this problem is:\n\n$$\\max_{\\boldsymbol{d}_t} \\min [ \\nabla   \\ell_u^1(\\boldsymbol{\\theta}_t) \\cdot \\boldsymbol{d}_t ,\\nabla   \\ell_u^2(\\boldsymbol{\\theta}_t) \\cdot \\boldsymbol{d}_t ]- \\frac{1}{2} \\|\\|\\boldsymbol{d}_t\\|\\|^2 \n\\text{ s.t. }  \\nabla   \\ell_r(\\boldsymbol{\\theta}_t) \\cdot \\boldsymbol{d}_t \\geq - \\varepsilon_t$$\n\nThe solution to this involves applying unilateral gradient surgery to each unlearning gradient ($\\nabla \\ell_u^1$ and $\\nabla \\ell_u^2$) and then aggregating the resulting gradients via a minimum norm operation, similar to MGDA [3]. The process is the same for more concepts. Importantly, this multi-concept explicit surgery can also utilize the approximation method from our paper to avoid the cost of multiple backpropagations. We will add the above discussion in the next version.\n\n[1]Lin, Shen, et al. \"GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2] Qu, Youyang, et al. \"Learn to Unlearn: A Survey on Machine Unlearning.\" *arXiv preprint arXiv:2305.07512* (2023).\n\n[3] Sener, Ozan, and Vladlen Koltun. \"Multi-Task Learning as Multi-Objective Optimization.\" In *Advances in Neural Information Processing Systems 31* (NeurIPS), 2018.\n\n[4] Lu, Shilin, et al. \"MACE: Mass Concept Erasure in Diffusion Models.\" In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.\n\n[5] Gandikota, Rohit, et al. \"Unified Concept Editing in Diffusion Models.\" In *Proceedings of the Winter Conference on Applications of Computer Vision (WACV)*, 2024."}, {"review_id": "Z6OJHE8KJU", "reviewer": "Reviewer_7Sf9", "summary": "Machine unlearning aims to remove the memory of specific data from the pre-trained model, and there is a trade-off between increasing the effectiveness of unlearning and maintaining the performance of the original model. One way to mitigate this is to employ a multi-objective optimization, but existing methods are inadequate for unlearning. To address this issue, the paper formulates a constrained optimization problem that aims to optimize the unlearning objective under the constraint of a bounded increase for utility loss and proposes an efficient method based on gradient surgery to solve this problem.", "questions": "1. Could you clarify the precise definition of \"pure linearization\" used in Table 5 and provide a citation or detailed explanation of \"RL\" in Section 4.3?\n2. How sensitive is the performance to the value of the error tolerance $\\varepsilon$?\n3. Have you analyzed why the proposed method becomes less cost-effective specifically in the random forgetting scenario?\n4. Why is the evaluation process different from existing studies?", "limitations": "The authors discussed the limitations in the appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. This paper accurately provides why multi-objective optimization cannot be directly applied to the challenges of machine unlearning.\n2. Each proposition is thoroughly proven in the appendix.\n3. The paper presents comprehensive experimental results, and the image generation outcomes clearly illustrate the effectiveness of the proposed method.\n\n## Weaknesses\n1. While existing unlearning studies [1, 2] are evaluated on datasets with diverse classes such as CIFAR-100, Tiny-ImageNet, and ImageNet-1k, the paper only conducts experiments on relatively simple datasets with 10 classes, such as CIFAR-10 and Imagenette. As a result, the evaluation appears insufficient to fully validate the generalizability and scalability of the proposed method.\n2. In Section 4.3, the definition of \"pure linearization\" is missing. There is no citation or clear description for \"RL\". Table 5 shows the proposed method takes more computation time than \"pure linearization\". If \"pure linearization\" refers to \"linear weighting,\" then it raises concerns about the effectiveness of the proposed method, which claims to achieve comparable computational efficiency to \"linear weighting\".\n3. While it is understandable that the hyperparameter \"error tolerance\" $\\varepsilon$ influences empirical performance, there is no sensitivity analysis of the performance to $\\varepsilon$. This lack of analysis makes it difficult to assess the robustness of the proposed method.\n4. Table 6 shows that the proposed method incurs a significantly higher computational cost compared to GA in the random data forgetting scenario, while achieving only marginal performance gains. This raises questions about its cost-effectiveness.\n5. While existing methods [1, 2] evaluate unlearning performance by quantifying the gap between an unlearned model and a model retrained from scratch without data subject to unlearning, this paper evaluates performance using raw accuracy values. Given that the objective of unlearning is to obtain a model equivalent to one trained without using the data, it would be more appropriate to evaluate performance based on the gap from the retrained model trained only on the retained data.\n\nMinor:\n\n6. Some highlights in Tables 5, 6, and 7 are incorrect.\n7. There is a typo in line 761 of Appendix F: \"$varepsilon_t$\" should be \"$\\varepsilon_t$\".\n\nReferences\n\n[1] Fan et al., Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation, In Proc. ICLR, 2024.\n\n[2] Jia et al., model sparsity can simplify machine unlearning, In Proc. NeurIPS, 2023.", "was_revised": true, "final_justification": "The authors’ rebuttal addresses most of my concerns, and I have accordingly increased my initial score.", "rebuttal": "We would like to thank reviewer 7Sf9 for the insightful comments. We address the concerns below.\n\n**W1. Datasets are relatively simple...the evaluation appears insufficient to fully validate the generalizability and scalability of the proposed method**\n\nGood suggestion! Our study deliberately spans multiple data regimes—natural-image classification, generative diffusion, style/instance manipulation, and safety filtering. We also evaluate on different backbones of Resnet, VIT, DDPM and Stable Diffusion. Each results are averaged over multiple independent trials in more than one different code base implementation. Therefore, we believe that such diverse experimental settings could already be able to prove the generalizability and scalability of the proposed method.\n\nMeanwhile, we tested the performance of Resnet 18 unlearning on Cifar 100, random data forgetting 10%.\n\n| **Methods** | **UA** | **RA** | **TA** | **MIA** | **Avg. Gap 4** | **Avg. Gap 2** |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Retain | 26.47 | 99.97 | 74.13 | 51.00 | 0.00 | 0.00 |\n| FT | 2.42 (24.05) | 99.95 (**0.02**) | 75.55 (1.42) | 11.04 (39.96) | 16.36 | 20.69 |\n| RL| 55.03 (28.56) | 99.81 (0.16) | 70.03 (4.09) | 98.97 (47.97) | 20.20 | 26.03 |\n| GA | 3.13 (23.34) | 97.33 (2.64) | 75.31 (1.18) | 7.24 (43.76) | 17.73 | 22.47 |\n| IU | 3.18 (23.29) | 97.15 (2.82) | 73.49 (0.64) | 9.62 (41.38) | 17.03 | 21.01 |\n| BE | 2.31 (24.16) | 97.27 (2.70) | 73.93 (**0.20**) | 9.62 (41.38) | 17.11 | 20.79 |\n| BS | 2.27 (24.20) | 97.41 (2.56) | 75.26 (1.13) | 5.82 (45.18) | 18.27 | 23.16 |\n| l₁-Sparse | 10.64 (15.83) | 96.62 (3.35) | 70.99 (3.14) | 22.58 (28.42) | 12.68 | 15.78 |\n| SalUn | 27.53 (**1.06**) | 97.00 (2.97) | 67.79 (6.34) | 70.79 (19.79) | 7.54 | 13.07 |\n| SalUn-soft| 24.24 (2.23) | 98.95 (1.02) | 70.48 (3.65) | 79.13 (28.13) | 8.76 | 15.89 |\n| **EUPMU(ours)** | 13.53 (12.94) | 97.02 (2.95) | 71.90 (2.23) | 51.02 (**0.02**) | **4.53** | **1.12** |\n\n\nThese results are consistent with the conclusions presented in the paper. We will include results on larger-scale datasets, such as Tiny-ImageNet and ImageNet-1k, in a future version of the paper.\n\n\n**W2 \\& Q1.Define \"pure linearization\"  and \"RL\"...it raises concerns about the effectiveness of the proposed method, which claims to achieve comparable computational efficiency to \"linear weighting**\n\nThank you for pointing this out! \"RL\" refers to Random-Labeling as cited in Appendix D.1, and pure linearization indeed refers to \"linear weighting\".\n\nThe introduction of an approximation to the gradient method for the Lagrangian (equation 4) avoids the need to explicitly solve for the two objectives. However, the computation of $\\ell_r (\\theta_{t+1})$ (equation 8) introduces an additional forward pass. This results in a slight increase in computational complexity compared to linearization, although the computational cost is still significantly lower than that of standard backpropagation. This explains the slight increase in computational cost shown in Table 5.\n\nHowever, this additional forward pass can be eliminated by reusing the value of $\\ell_r (\\theta_{t+1})$ computed during the forward pass of the next iteration (*t*+1). While this may lead to a mismatch between the data batches at step *t* and *t*+1 during stochastic training, it does not affect the model's convergence due to the unbiased nature of the stochastic process. Our latest experiments show that this modified approach (see EUPMU (modified)) achieves similar results comparable to EUPMU (previous) in Table 5, with a computational time on par with linearization.\n\n\n| Methods | UA | RA | TA | MIA | Avg. score | RTE |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Linearization  | 14.11  | 90.92 | 86.05  | 20.16  | 14.325 | **15.7** |\n| **EUPMU (previous)**  | **0.49**  | **99.88** | 94.42  | **1.75**  | **1.985** | 18.4 |\n| **EUPMU (modified)**  | 0.51  | 99.79 | **94.45**  | 1.79  | 2.015 | **15.7** |\n\nWe will discuss this more lightweight computational approach and provide sufficient experimental validation in a future version of the paper.\n\n**W3 \\& Q2. Error tolerance $\\varepsilon$ sensitivity...This lack of analysis makes it difficult to assess the robustness of the proposed method**\n\nBelow, we provide an expanded ablation study error tolerance $\\varepsilon$.\n\n| $\\varepsilon$ | Retain Loss | Forget Loss |\n| :---: | :---: | :---: |\n| 0.00  | 0.052  |   **2.25** | \n| 0.10 | 0.035  | 2.53  | \n| 0.25 | 0.031  | 2.92  | \n| 0.50 | 0.029  | 3.58  | \n| 1.00 | **0.028** | 4.41  | \n\nWe observe that as $\\varepsilon$ ranges from 0 to 1, our algorithm smoothly traces the Pareto front. This confirms that the utility-unlearning tradeoff is controllably adjustable, allowing practitioners to select a balance suitable for their specific needs. This finding is in direct alignment with our statements in Remark 3.3 and Remark 3.6.\n\nRegarding the tuning of $\\varepsilon_t$, this process is not burdensome. As guided by our theoretical results in Remark 3.3 and Theorem 3.7, one can effectively narrow the search range based on the magnitude of the loss function, which minimizes practical tuning effort. We will add further experiments and discussion to clarify this in the revised manuscript.\n\n**W4 \\& Q3. Cost‑effectiveness vs. GA in Table 6**\n\nThe running time comparison in Table 6 requires careful context. Fundamentally, the per-iteration computational cost of EUPMU (with the optimizations from our W2 response) and Gradient Ascent (GA) is the same. The lower time for GA is an artifact of its training instability: it must be early-stopped at 20% of epochs to prevent gradient explosion, which would otherwise cause a catastrophic drop in Test Accuracy (TA).\n\nIf run for an equal number of epochs, EUPMU would have a similar runtime to GA but achieve a higher TA due to better utility preservation.\n\nMoreover, GA's result in Table 6 is a special case. It performs poorly on all other datasets (e.g., Table 7). This highlights the advantage of our method, which provides a superior trade-off, delivering more effective unlearning and utility preservation without a true increase in computational cost.\n\n**W5 \\& Q4. Evaluate gap‑to‑retrain metric.**\n\nFrom a practical standpoint, directly optimizing for raw accuracy is more beneficial for real-world applications. In practice, the results of a fully retrained model (the \"retrain oracle\") are often unavailable. As long as the model's utility and data privacy metrics are guaranteed, its practical effectiveness is ensured. In our experiments, the unlearning process further fine-tunes the model on the retained data via the retain_loss. This can lead to performance that is better than retraining from scratch, especially on simpler tasks (like Table 6). In such cases, a comparison based on the \"gap-to-retrain\" metric would not be fair. Therefore, we chose to present our results using raw accuracy.\n\nIf the goal is to minimize the gap to the Retrain Oracle, EUPMU still outperforms other methods. We benchmarked against GDR-GMA [1] and Learn to Unlearn [2]—strong baselines noted by reviewers and established as state-of-the-art in their publications. The following table presents the results for class-wise unlearning with ResNet-18 on CIFAR-10:\n\n| Method | Retain Acc | Forget Acc | Test Acc | MIA  | Avg Gap  |\n| --- | --- | --- | --- | --- | --- |\n| Retrain Oracle | 99.42 % (0.00) | 0.00 % (0.00) | 94.08 % (0.00) | 0.000 (0.000) |   0.00 |\n| GDR-GMA  | 95.30 % (4.12) | 0.00 % (**0.00**) | 88.70 % (5.38) | 0.000 (0.000) |   2.38 |\n| Learn to Unlearn | 98.98 % (**0.44**) | 0.76 % (0.76) | 92.00 % (2.08) | 0.000 (0.000) |   0.82 |\n| **EUPMU (Ours)** | 98.75 % (0.67) | 0.09 % (0.09) | 92.18 % (**1.90**) | 0.000 (0.000) |   **0.67** |\n\nAs shown in the above gap‑to‑retrain results (and also the table in the response of W1), when we optimize the hyperparameters for gap-to-retrain, EUPMU still performed the best compared to other methods as well.\n\n[1]Lin, Shen, et al. \"GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2] Qu, Youyang, et al. \"Learn to Unlearn: A Survey on Machine Unlearning.\" *arXiv preprint arXiv:2305.07512* (2023)."}]

Author rebuttal:

Meta-review: The paper proposes a new framework for utility-preserving machine unlearning. The central claim is that existing approaches either suffer from severe utility loss (linearization methods) or from insufficient unlearning and efficiency issues (multi-objective optimization). To address this, the authors formulate the problem as a constrained optimization and derive a gradient surgery algorithm. The method is supported with theoretical guarantees, including Pareto optimality/stationarity in both convex and non-convex cases. This formulation is novel in explicitly treating MU as a constrained optimization problem with provable utility control.

Although several reviewers raised questions regarding novelty compared to gradient conflict correction methods, they are addressed by authors properly during the rebuttal. One reviewer (DBBQ) who had raised the above concerns did not participate in the rebuttal/discussion phase. However, given that their specific points were thoroughly addressed and positively acknowledged by other reviewers who had raised comparable issues, the area chair is satisfied that these concerns have been sufficiently resolved.

Overall, the paper makes a significant and theoretically solid contribution to the machine unlearning literature. Most reviewers recommend acceptance, and the additional evidence provided during rebuttal further strengthens the case for this paper.
Decision: Accept (poster)