DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
Authors: Xuyang Zhong
Haochen Luo
Chen Liu
Venue: NeurIPS 2025 poster ()
Keywords: machine unlearning
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Existing machine unlearning (MU) approaches exhibit significant sensitivity to hyperparameters, requiring meticulous tuning that limits practical deployment. In this work, we first empirically demonstrate the instability and suboptimal performance of existing popular MU methods when deployed in different scenarios. To address this issue, we propose Dual Optimizer (DualOptim), which incorporates adaptive learning rate and decoupled momentum factors. Empirical and theoretical evidence demonstrates that DualOptim contributes to effective and stable unlearning. Through extensive experiments, we show that DualOptim can significantly boost MU efficacy and stability across diverse tasks, including image classification, image generation, and large language models, making it a versatile approach to empower existing MU algorithms.

DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
Xuyang Zhong
Haochen Luo
Department of Computer Science City University of Hong Kong xuyang zhong@my. cityu. edu..eu.... .k
Department of Computer Science
City University of Hong Kong chester hc. luo@my cityu. edu....d.... hk
Chen Liu
Department of Computer Science City University of Hong Kong chen. liu@citityi tyu. edu. hk
Abstract
Existing machine unlearning (MU) approaches exhibit significant sensitivity to hyperparameters, requiring meticulous tuning that limits practical deployment. In this work, we first empirically demonstrate the instability and suboptimal perfor- mance of existing popular MU methods when deployed in different scenarios. To address this issue, we propose Dual Optimizer (DualOptim), which incorporates adaptive learning rate and decoupled momentum factors. Empirical and theoretical evidence demonstrates that DualOptim contributes to effective and stable unlearn- ing. Through extensive experiments, we show that DualOptim can significantly boost MU efficacy and stability across diverse tasks, including image classification, image generation, and large language models, making it versatile approach empower existing MU algorithms. Codes are available at hhttps://iithub. comm/CttUU MLO//ualOptmm
1 Introduction
Recent advancements in machine unlearning (MU) research have established as a crucial technique for utilizing pretrained models while addressing various trustworthy challenges in various applica- tions [1] These MU methods have two major categories: exact MU [3] and approximate MU [4]. Exact MU requires retraining model on dataset excluding forgetting samples from scratch, which computationally prohibitive for large-scale models and datasets. Therefore, we study approximate MU methods in this work, with focuses their efficacy and stability.
7 g g aim to maximize loss on forget samples while preserving performance retain samples. While effective, these methods exhibit significant sensitivity to hyperparameter choices, with optimal configurations requiring meticulous tuning that varies substantially across datasets and forgetting scenarios. This dependency complicates their practical deployment. As illustrated in Figure [ we empirically demonstrate that state-of-the-ee MU approaches [6] 7 (8) cxhibit suboptimal performance instability. These limitations highlight the critical need for techniques to improve the effectiveness and stability of MU methods.
Most existing approximate MU methods [5]
In this work, we first reveal the unstable performance of existing MU methods when they are applied in different scenarios. Considering the need optimize two objectives on the forgetting
CCoressponiigg Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

WRAT SCRLO
SCILE
SCRUR
S0R0
65 100 108 350
160
400 5000 D80
180
20
800 182
1100
140
22
400
109
(a) Forget Accuracy (FA) (b) Retain Accuracy (RA)
(c) Test Accuracy (TA)
(d) MIA
Figure I: Unlearning process of MU baselines. SFRon 7. SalUn
6 and SCRUB g are adopted as
3.1 All results are obtained from unlearning 10% random subset of CIFAR-10 on ResNet-18. The solid lines and shadows denote the mean and standard deviation across trials with different random data. The hyperparameters of different methods are selected based on minimizing the averaging gap between retraining and them across 5 trials. The red dashed lines denote the final performance of retraining as a reference.
the baselines. The metrics are those mentioned Sec.
and the retaining samples, we then propose Dual Optimizer (DualOptim) to address this challenge. Specifically, we employ an optimizer with adaptive learning rate, such as Adam [T] 11 to optimize
the forgetting objective, while utilizing a separate optimizer for the retaining objective. Compared
with existing methods, DualOptim decouples the momentum terms for the two objectives by using and updating them separately. In addition, DualOptim is plug-and-play solution that can be easily integrated into existing MU algorithms.
Our theoretical analysis validates that decoupling momentum terms in two separate optimizers can help decrease the variance of model parameters, indicating more stable performance. Through comprehensive experiments, we validate the effectiveness of DualOptim in improving the performance of existing MU methods across various tasks, including image classification, image generation, and large language models. Our proposed technique is generic and capable of pushing the state-of-the-art performance on multiple tasks. We summarize the contributions of this paper as follows:
1. We introduce Dual Optimizer (DualOptim) to enhance the efficacy and stability of MU methods by incorporating an adaptive learning rate and decoupled momentum. DualOptim can be seamlessly integrated into existing MU algorithms.
2. We provide empirical and theoretical analyses to demonstrate the contribution of DualOptim in improving unlearning performance and stability.
3. Comprehensive experiments generation, and large language models validate the effectiveness of DualOptim in boosting and stabilizing the performance of MU methods.
diverse scenarios such image classification, image
2 Related Works
Machine Unlearning (MU). MU targets the need to remove specific data influences from pretrained models
13
while complying with privacy requirements tied differential privacy B 15
16
Initially developed on linear classifiers with convex loss objective functions, exact allowed precise data removal under privacy budgets to counter privacy attacks. However, exact MU cannot be applied to deep neural networks due to their non-convex loss functions and prohibitive computational cost for full retraining. In this context, approximate MU 21 was proposed to address this issue by fine-tuning the model to achieve the forgetting effect. Despite improved efficiency, approximate MU may cause catastrophic performance declines on retaining data Recent methods incorporate fine-tuning 66, 23) sparsity regularization (24), knowledge distillation g (25), saliency map 66.77 | and alternative updating [7] to better balance efficient forgetting and model utility. However, as illustrated in Figure T these methods generally exhibit suboptimal performance or high hyperparameter sensitivity, underscoring the necessity to develop a method to enhance efficacy and stability during unlearning.
17
8
MU 3
15
21 [22
MU for Multiple Tasks. Besides classification,, MU has broad applications for multiple tasks. For example, recent text-conditional image generation models have showcased their ability to produce images that closely align with textual descriptions 26
27 28 29]. However, significant security and
privacy concerns have been raised
30 31
necessitating the application of MU methods to these

---

models. While early works [30 32 33 34 focuses on concept deletion in diffusion models, recent studies 7
351 improve their performance and propose methods applicable to more general image generators. Another notable example is large language models (IJ.Ms), which have demonstrated remarkable capabilities
37 but also face privacy and copyright issues like retaining unauthorized To address these concerns, MU methods have been employed to fine-tune 43 to effectively and efficiently achieve data forgetting. However, all these methods struggle to achieve balance between model utility and forget effectiveness. In addition, intensive hyperparameter tuning is expected for optimal performance, bringing challenges for practitioners.
36
content [38
39
40
LLMs 22
9
42
In summary, MU has broad applications across various tasks. However, there are some common issues with existing methods, including high trade-offs. In this work, we propose DualOptim as a generic solution to enhance the efficacy and stability of MU across different tasks.
sensitivity and the
3 Methodology
3.1 Preliminary
Let D = {2,) denote the training set for pretraining. The subset of the training set we aim to forget during unlearning is known as the forget set D, C CD, and its complement, DD D DD is the retain set. In the context of MU, we denote the parameters of pretrained model as 0, E Rd which is trained on D. Consistent with previous studies 6
7 21 24], Retraining is considered the gold standard for MU, where the model parameters 0, E R are trained from scratch on D,.. However, retraining is computationally intensive or even infeasible, especially for large models and datasets. This poses significant challenge in the practical applications of MU.
We focus on approximate MU in this work, its primary objective to obtain an unlearned model, referred to as 02 E Iidd from 0, on Dj and D, so that it can serve an accurate and computationally efficient alternative to the retrained model 0,. Mathematically, MU aims to solve the optimization problem: ming L (D,, 0) + L. (D,, 0), where L and C, denote forget and retain loss objective functions, respectively. In practice. C, is usually the same as the loss objective function in pre- training and Ly is the opposite, so MU aims to improve the performance on D, while degrading the performance on Dy. To avoid confusion, we call (, f. 0). (D, 0) that we aim to forgee loss and retain loss, respectively. In addition, many MU algorithms 66 7 8 update 0 by alternately optimizing (D 0) and (D,, 8) for better performance. The generic pipeline of MU is presented in Algorithm|| For notation simplicity, denote the gradients of the forget loss and the retain loss, respectively. In mini- updates, we use 9, and 9. denote the corresponding stochastic gradients.
use g V @) and g.- (e,,(D,, 0) to
The numerical analysis in this section based on classification tasks. We adopt the same evaluation schemes as M7 That is, we use the accuracy on the forget set (FA), the retain set (RA), test set (TA) and the success rate of membership inference attack (MIA) [44) model utility and the forgetting effectiveness. A competitive unlearning method should have stable performance and small gaps with retraining in all these four evaluation schemes.
Pon on the forgetting set to indicate
In the following subsections, we first illustrate the challenges associated with MU and propose corresponding approaches to address them. Ultimately, we introduce DualOptim, which integrates these proposed techniques to enhance both efficacy and stability in MU.
3.2 Adaptive Learning Rate Enables Stable Forgetting
Despite the effectiveness of existing MU methods, achieving satisfactory unlearning performance of- ten relies on carefully selecting hyperparameters. The optimal hyperparameters can vary significantly across different forget sets, complicating the practical applications of MU algorithms. As shown in Figure (a)-(d). we use the same hyperparameters for SFRon [7], the best pest-performed baseline in our evaluation, with the default SGD optimizer for trials of different randomly sampled forget sets. We defer more results of other existing methods to Appendix C.2) Despite being intensively tuned, the unified hyperparameters exhibit quite unstable performance across different forget sets during unlearning. This underscores that a hyperparameter configuration effective for certain forget sets may
We follow the implementation in and adopt the entropy of the output probabilities as the metric in MIA.
3

---

500
300
150
Qualppin
Dualdsim
Dualattn
Retrol
Adtrsin
Itemtion
Heradon
Itenstinn
Ieration
(a) Forget Accuracy (FA) (b) Retain Accuracy (RA) (c) Test Accuracy (TA)
(d) MIA
Figure 2: Unlearning process with different ablations of the proposed method. All results obtained from unlcarning 10% random subset CIFAR-10 10 by SFRon (7] on ResNet-18. (a)-(d) The metrics those mentioned in Sec. 3.1 The red dashed lines denote the final performance of retraining
reference. The solid lines and shadows denote the mean and standard deviation across trials with different random forget sets.
SGD Forget 5G0 Betain
Adam Adam Bebain
Dus Oetim Frrget
80-
03-
Galm Beta
Cradientt Norm
Gradient Norm
Gradient
60-
02-
426
100 Ado 1997 Iteration
1297
520
41C
1200
Reration
iteration
(a) SGD
(b) Adam
(c) DualOptim (Ours)
Figure 3: Norms of stochastic forget gradient 9j and stochastic retain gradient 9r using different optimizers. All results are obtained from unlearning 10% random subset of CIFAR-10 by SFRon on ResNet-18. (a)-(e) The curves are obtained using SGD, Adam, DualOptim, respectively.
not be suitable for others. When dealing with a new forget set, even if is sampled in the same way, the need for precise hyperparameter tuning can pose challenges for practitioners.
Figure (a) demonstrates the magnitude of the gradients the forget and the retain set when using SGD optimizer. We can clearly see that the gradients on the forget set have significantly larger magnitudes with substantial variations compared with the ones on the retain In this context, is crucial to adaptively adjust the learning rate to handle different gradient magnitudes. Therefore, we use optimizers with preconditioners, such as Adam [IO [1] evade tricky hyperparameter tuning for the learning rate. Specifically, Adam employs adaptive learning rates based on the historic gradient magnitudes, making suitable to handle large gradient magnitude variations during unlearning.
The observations Figure 2 (a)-(d) and Figure 3 (b) suggest that Adam provides more stable performance and induces smaller gradient norm across different forget sets. However, despite enhanced stability, the noticeable performance gap with retraining, especially in the metric of membership inference attack (MIA), indicates that Adam only achieves suboptimal unlearning efficacy. We need a mechanism to improve performance while ensuring the stability of the method.
3.3 Decoupled Momentum for Enhanced Stability in Machine Unlearning
Inspired by the disparities in the magnitudes of ğj and gr. along with their non-positive correlation observed in Figure 3 and 4 we introduce decoupled momentum, which employs two separate momentum terms dedicated to the forget loss Ly and the retain loss L,, respectively. This approach ensures that the momentum update for the forget loss remains unaffected by the gradients of the retaining loss, and vice versa. By decoupling the momentum terms in this manner, we can more effectively optimize the distinct processes of forgetting and retaining, leading to enhanced stability and performance. As illustrated in Figure (a)-(d) (represented by the green lines) and Figure (c). the combination of adaptive learning rate with decoupled momentum not only stabilizes the unlearning process also leads to improved unlearning efficacy. Note that as presented in Table and 6 of Appendix C.2 although other methods like SalUn 6 and SCRUB g perform more stable than SFRon, they only achieve suboptimal unlearning performance. Nevertheless, our method can also improve their performance.

---

Besides empirical findings, we provide theoretical analysis to elucidate how decoupled momentum contributes to stability. Before the detailed analyses, we first establish the following assumptions.
Assumption 3.1. (Stochastic Gradient Condition) For all time steps = ...., T the stochastic gradients of the forget loss Jf., and retain loss Jr.t satisfy:
(I) 02) and Dr., (D,,, are the full-batch gradients with model where Jf. = V,,
g = 9fft + Cftt gr = gr,t + Cr,ts
parameter 0g at the time stamp t. ES and E,,, are batch noises with zero mean and a bounded variance: there exists a minimal o² 2 such that Var(es,)) S o², Var(e,,) S o² for all
Assumption 3.2. (Correlation Bounds) The correlation between the stochastic gradients from the same function in different time steps is bounded while the correlation between stochastic gradients from different functions is non- non-positive. That to say, Br E [0. 1] such that:
vt1 * ,.t. pl : M...) S T, P( Ir..) S
Vt1, 12, Plā Ir,t.) - o(T) =
(2)
The assumption 14,, 12, : I o(T)
motivated by the observations in Figure 4 of Appendix C.T and for sake of notation simplicity. Our analyses can be easily extended to the cases if we use a small constant to bound this correlation.
Assumption 3.3. (Lipschitz Smoothness) The loss functions L and C, are both L-smooth:
v0, 02, V Ly 01) Vo, (,,,,(D,,,) 02)1 & L| 0211:
(3)
101, 82. Vo, C. (D,, 01) - (D,, 02)1l S I:10] 0211-
We assume the same Lipschitz constant for Cj and L, because (1) Dy and D, are from similar distributions; (2) Ly and L, are usually opposite functions.
We now consider the SGD update scheme with a shared or decoupled momentum factor. Since alternately update the parameters based the stochastic gradients from Ly and fr, we {(mf. m )o to denote the momentum factors after using the gradients from the forget loss and the retain loss, respectively, for time stamp when we using the shared momentum. Similarly, T-1 denote the momentum factors when using the decoupled momentum. F.t (T to represent the corresponding updated parameters. We use E [0, 1] to denote the momentum factor and as the learning rate, then the update schemes for shared and decoupled momentum factors are shown follows:
use {(m f,t m.)))=
We use {(0S,
f.t. 0 ( (OP,,
(Shared Momentum)
m
am 9 r 1-1-111 + S
g1.0
f 0,, nmm
$
m am ftt + gr.t
0 ff, 1.1e
(5)
(Decoupled Momentum)
m
D
D cmm + f.
f.t 0D 0
-ym
m
amm + Tr.ts
r, 0f.
-mm
Here, we use [(G D')) and ((PP. ((P....P.))) to denote the stochastic gradients during un- learning in the case of shared momentum and the decoupled momentum, respectively. Based on Algorithm I] we update the parameters by gradients from the forget loss and the retain loss al- ternately. Therefore, we have 9¹, gh, = (O,)) and Jf., T.. Ii, P.P are their stochastic variants.
(6 1) P.. (oL,(0P,.)) g (0S,
When using decoupled momentum factors, the momentum factors (m f.o: m D (m". m. m....... D mentum factor, the factor is updated as a sequence {m² m m f.1 m........ m J.T-1 - m 11 When initialization, we let 05 . 0, and m m m - 0.
f.1 m D f
are two independent sequences. By contrast, when using shared mo-
We focus on the variance of the parameters in two different update schemes in ( both of which iteratively calculate the gradients random variables. Therefore, first estimate the variance caused by gradient calculation as in the following lemma.
Lemma 3.4. (Variance of Gradients) If the loss function Lipschitz smooth with a constant L., and Var (0) S G. then we have Var (V)(()) < L'og.
5

---

The proof and discussions are deferred to Appendix A.1 We can use Lemma 3.4 to derive the maximum variance of model parameters for both update schemes ( in following theorem.
Theorem 3.5. (Variance Bound Comparison for Decoupled vs. Shared Momentum) For the update schemes indicated in ( using the same hyperparaaetters (n, a),, and we use Var(-) to denote the maximum variance of a variable, if the function L,, C. and the stochastic gradient {(]); P -
(.P....PP.)) satisfy Assumption
3.1 3.2
and 3.3 then
VI, Var(0
Var(O Var(O S Var(O,;,),
The proof is deferred to Appendix AA.2] Theorem 3.5 formalizes that decoupled momentum induces smaller worst-case parameter variance compared to shared momentum, demonstrating that decoupled momentum theoretically enhances the stability of unlearning. Additionally, we prove that decoupled momentum can also induce smaller worst-case variance of downstream metrics in Appendix A.3
3.4 Dual Optimizers for Machine Unlearning
Algorithm Machine Unlearning with Shared Optimizer Dual Optimizers
1: Input: Model: Se: Forget set: Dy; Retain set: "D; Iterations for outer loop: To: Iterations for forgetting: Ty; Iterations for retaining: T,: Step sizes: 7 775. 77r
2: Optim is the same optimizer as in pretraining with step size 17-
Optim, , is Adam(0 ny). Optim, is the same optimizer in pretraining with step size T)--
3: for t = ...., To do
4: for = 1, ...,, T'y do
5:
Fetch mini-batch data from the forget set B, ~ Dj Calculate the forget loss Ly on By and get the gradient
6:
7: Use Optim Optim, to update 0
8:
end for
9:
for
1...., T. do Fetch mini-batch data from the retain set B, D,
10:
11: Calculate the retain loss C, on B, and get the gradient
12: Use Optim Optim, update
13:
end for
14:
end for
15:
Output: Model fo
We incorporate the findings of the subsections above and propose Dual Optimizers (DualOptim) to enhance the efficacy and stability of MU. Specifically, we utilize two distinct optimizers during the unlearning process: Adam for forgetting and the default optimizer (e.g., SGD) for retaining. On one hand, we use the same optimizer as in pretraining to maintain the performance on the retain set and use the optimizer with adaptive learning rates to handle the large gradient variation for the forget loss. On the other hand, we decouple the momentum factors and use two distinct optimizers to boost the algorithm stability during unlearning. Notably, DualOptim a plug-and-play approach and can be integrated in existing MU algorithms that minimize forgetting and retaining objectives alternately. The detailed pseudo-code to compare the pipeline of MU with a shared optimizer and DualOptim is presented in Algorithm[1 [ with unique segments for shared and dual optimizers highlighted in red and green, respectively. Algorithm T presents a general framework, L, and L, are specified by different concrete MU algorithms.
4 Experiments
In this section, we conduct extensive experiments to evaluate our method for different applications, including image classification, image generation, and large language models. The results demonstrate that our method can enhance the efficacy and stability of multiple MU methods, achieving new state- ofethe-art performance across diverse scenarios. We conduct ablation studies for further analysis.

---

4.1 Random Subset Unlearning in Image Classification
Table I: Performance summary of MU methods for image classification. Experiments are conducted on (a) 10% random subset of CIFAR-10 using ResNet-18 and (b) 10% random subset of TinyIma- geNet using Swin-T. All results are presented as mean and standard deviation across trials with different random forget data. Performance gaps with RT are indicated in blue. The average gap (Gap) and average standard deviation (Std) metrics are calculated by the average of the gaps and standard deviation measured in FA, RA, TA, and MIA, respectively. All the numbers are in percentage.
(a) CIFAR-10 Random Subset Unlearning (10%)
Method
FA
RA
TA
MIA
Gap +
Std
RT
94.61 20.46 (0.00)
100.00 (0.00)
94.25 (0.00)
76.26 (0.00)
0.00
0.30
FT
99.16 (4.55)
99.84
(0.16)
(0.15) 94. 24.1020.09
88. 38.77 20.38 (12.51)
4.34
0.16
GA
98.7610.39 (4.15)
19.1010.90
(0.90)
93.89
(0.36)
92.58 1...0... 55 (16.32)
5.43
0.44
RL
97.1910.21 (2.58)
99.67 10.08 (0.33)
94.0330.27 (0.22)
68. 19...5 (8.43)
2.80
0.38
SCRUB
92.88 (1.73)
99.62.0.10 10 (0.38)
03.54-0..22 (0.71)
82.78 (6.52)
2.33
0.36
+DualOptim
94,90.0.12 (0.29)
99. 99.52...00 (0.48)
00.50
(0.75)
78.26 (2.00)
0.88
0.38
SalUn
96,99 10.31 (2.38)
99.4010.22 10.28 (0.60)
93.8410.30
(0.41)
65.76
(10.50)
3.47
0,50
+DualOptim
95.47 (0.86)
99.06 00.4 (0.60)
92.47=0.20 (1.78)
76. 14 (0.12)
0.93
0.35
SFRon
94.67 +3.03 (0.06)
99 (0.17)
=3.98=0055 (0.27)
77.80+5.01 (1.54)
0.51
2.33
+DualOptim
94.6921.13 +1.13 (0.08)
99.92 (0.08)
94. =4.11 (0.14)
77.77 (1.51)
0.44
0.66
(b) TinyImageNet Random Subset Unlearning (10%)
Method
FA
RA
TA
MIA
Gup J
Std 4
RT
85.29 (0.00)
99.55_0.08 (0.00)
85,4920.15 15 (0.00)
00. (0.00)
0.00
0.12
FT
96.50 (11.21)
98.23-o.0s (1.32)
82.67 (2.82)
79.85
(10.55)
6,48
0.13
GA
90.02+ (4.73)
90.84-3.29 (8.71)
75.64_2.07 (9.85)
78.97
(9.67)
8.24
2.82
RL
94.66 26 (9.37)
98.02_0.14 14 (1.53)
82.73_0.27 (2.76)
54.45=1.04 (15.15)
7.13
0.43
SCRUB
97 (12.51)
98.13
(1.42)
82.64
(2.85)
79.62 41 (10.32)
6.78
0.21
+DullOptim
97.20+0.20 (11.91)
98.30 98.30 (1:25)
83,17=0..19 (2.32)
79, 79.10+00.6 (9.80)
6.32
0.28
SalUn
97.69+0.14 (12.40)
98.89=0.os (0.66)
84.02=0.32 32 (1.47)
60.87 (7.43)
5.49
0.37
+DualOptim
91.68 (6.39)
95.13_0 18 (4.42)
80.1620.34 (5.33)
72.48 20.88.0.30 33 (3.18)
4.83
0.28
SFRoa
90.41 (11.12)
98.95 22 (0.60)
83,40
(2.09)
70.40 (1.10)
3.73
1.16
+DualOptim
92.26 (6.97)
98.27
(1.28)
83.12 (2.37)
69.19 (0.11)
2.68
1.01
We start with random subset unlearning tasks in image classification, including using ResNet-18 18 on CIFAR-10 10 [46] and Swin Transformer-Tiny (Swin-T) [47] on Tiny TinyImageNet [48] Additional
45
results on CIFAR- 100 [46] and SVHN [4] are included in Appendix C.3 Consistent with previous
work 21 24 6 7 we regard Retrain (RT) as the gold standard of MU. Since our proposed DualOptim
is plug-and-play, we apply it to SCRUB (.) SalUn (6. and SFRon 7 to validate its efficacy and
stability. Additionally, we include three simple baselines, namely Fine-tune (FT) (50]. Gradient Ascent (GA) [21] and Random Label (RL) (22). for reference as well. Note that the hyperparameters of all evaluated methods are tuned to their optimal values by sophisticated search, and we defer the implementation details to Appendix B Besides the metrics mentioned in Sec. S.c. which includes FA, RA, TA and MIA, we follow the evaluation criteria in 6 to report the average gap (Gap) and average standard deviation (Std) to indicate the overall performance and stability, respectively. They are the average gap with RT and the average standard deviation among the results in FA, RA, TA, and MIA, respectively.
As illustrated in Table [] SFRon achieves the smallest average gap with RT, yet it exhibits the highest variability in performance across different random forget subsets. This suggests that the optimal hyperparameters vary significantly depending on the specific forget sets used. Despite being more stable, other methods yield suboptimal results in terms of unlearning efficacy. By integrating DualOptim into these MU algorithms, we observe notable enhancements in terms of both unlearning efficacy and performance stability. For example, in the task of unlearning 10% random subset from CIFAR- DualOptim reduces the average standard deviation of SFRon from 2.33% to 0.66%, while narrowing the average gap from 0.51% % to 0.44%, achieving the best performance among all. Additionally, DualOptim significantly narrows down the average gap with RT for both SCRUB and SalUn, with improvements of 1. 1.45% and 2.54%, respectively. Consistent results are observed in other datasets and model architectures as well, highlighting the widespread effectiveness of DualOptim in improving both the performance and stability of unlcarning processes.
7

---

4.2 Class-wise Unlearning in Image Generation
Class-wise unlearning performance on CIFAR-IO with DDPM and ImageNet with DiT. The best unlearning performance for each forgetting class is highlighted in hold for FA (in %) and FID. Note that the results of SA, SalUn and SFRon are those reported in [7].
Table 2:
CIFAR-10 Class-wise Unlearning
Method
Automobile
Cat
Dog
Horse
Truck
FA L
FID 4
FA
FID
FA
FID
FA
FID
FA |
FID
SA
0.00
23.56
14.20
21.34
8.60
21.19
0.00
21.13
0.00
29.04
SalUn
0.20
21.23
1.40
20.29
0.00
20.18
0.60
20.70
0.80
20.45
SFRon
0.00
20.70
7.40
18.44
0.20
18.89
0.00
19.93
0.00
20.61
+DO
0.20
19.72
1.00
19.36
0.00
18.58
0.00
18.91
0.00
17.26
ImageNet
Class wise Unlearning
Method
Cockatoo
Golden Retriever
White Wolf
Arctic Fox
Otter
FA
FID |
FA
FID
FA
FID
FA
FID
FA 4
FID
SA
0.00
348.75
0.00
298.97
0.00
45.89
0.00
393.91
29.8
321.21
SalUn
91.21
18.47
46.09
25.28
0.00
15.16
45,90
408.07
87.5
19.69
SFRon
0.00
13.59
0.00
17.76
0.00
23.28
0.00
16.12
0.00
16.43
+DO
0.00
17.46
0.00
14.63
0.00
14.72
0.00
14.91
0.00
14.55
Besides classification, we further evaluate our proposed DualOptim in class-wise unlearning tasks in image generation. Following the settings of 77. we conduct experiments using conditional DDPM [26] with the U-Net [51] CIFAR-I0 1- and the latent diffusion model (28) with Diffusion Transformer (DiT) 29 on ImageNet [48]. We apply our proposed DualOptim to SFRon (7]. which exhibit the
state-of-the-ars unlearning performance in image generation. We also include SA (35] and SalUn (6) as baselines for comparison. Note that we do not apply DualOptim to SA and SalUn due to their joint updating scheme and unstable performance on ImageNet. Nonetheless, SaIUn+DOO on CIFAR-10 10 in Appendix [C.4 The implementation details can be found Appendix adopt the accuracy of the unlearned models generated images
still present the results of
As for the evaluation metrics, on forgetting classes (FA) by a pre-trained classifier to indicate the forgetting efficacy and Fréchet Inception Distance (FID) 52 to assess the generation capability on the retained classes.
As shown in Table 2 and 3 our method generally enhances the fi- delity of generated images for retained classes while ensuring high forgetting efficacy and low variance. For exam- ple, it significantly reduces the FID of SFRon for the white wolf category from 23.28 to 14.63, with FA remain- ing at 0.00%. Importantly, these re- sults demonstrate that DualOptim is effective across various datasets and model architectures, making it suit- able for practical applications. To- gether with the findings in Section 4.1 these results highlight the effectiveness and generalizability of DualOptim in diverse computer vision tasks. Generated images from the unlearned model utilizing DualOptim can be found in Appendix C.7 The visualization again indicates that DualOptim achieves effective unlearning while maintaining the generation capability for the retained classes.
Table 3: Overall class-wise unlearning performance on CIFAR-10 with DDPM and ImageNet with DiT. The mean value and standard deviation of FA (in %) and FID among the classes evaluated in Table 2 are reported.
CIEAR-10
ImageNet
Method
FA
FID
FA
FID
SA SalUn
4,564a.86 23.25
5,96
281.754122.11
0.60
20.57 +0.37
54. 43..32
97.33
SFRon +DO
1.52 19.71 10.91
0.00
17.44
0.24 18.77
+0.86
0.00
15.25
4.3 Random Subset Unlearning in Large Language Models
Following the success of MU in vision tasks, there is rising interest of applying it to remove private or harmful information from large language models (LLMs). We conduct the experiments on Phi-1.5 553 and LLaMA 2 37 both are fine-tuned on TOFU dataset ( with 1.3B and 7B parameters, respectively. We include three untargeted unlearning methods (GA+GD, NPO+GD and ME+GD) and two targeted unlearning methods (DPO+GD and IDK+AP) 99] the baselines. We employ

---

Table 4: Performance comparison of different MU methods on TOFU-finetueed Phi-1.5 and LLaMA 2. The results include Model Capability (MC), Forget Efficacy (FE), and the average metric (Avg.) for forget 1%, 5% data, and 10% data.
Phi-1.5
Method
forget 1% data
forget 5% data
forget 10% data
MC
FE f
Avg. f
MC f
FE 1
Avg. 1
MC1
FE 1
Avg. f
GA+GD
0.4934
0.4493
0.4714
0,4360
0,5084
0.4722
0.4471
0.5246
0,4859
NPO+GD
0.2569
0.5682
0.4125
0,4940
0,4469
0.4705
0.4808
0.4382
0,4595
ME+GD
0.4944
0.3938
0.4441
0.4559
0,4480
0.4520
0,4594
0.4564
0.4579
+DO
0.4866
0.6913
0.5889
0.4676
0.8200
0.6438
0.5009
0.7732
0.6370
DPO+GD
0.2410
0.6831
0.4621
0,4105
0.6334
0.5219
0.3517
0.6302
0.4910
IDK+AP
0.4403
0.5723
0.5063
0.4800
0,5112
0.4956
0.4614
0.6003
0.5308
+DO
0.4221
0.7037
0.5629
0,4633
0.6974
0.5804
0.4422
0.7193
0.5807
LLaMA 2
Method
forget 1% data
forget 5% data
forget 10% data
MC
FE f Avg. f
MC f
FE
Avg. f
MC
FE 1
Avg. 1
GA+GD
0.6696
0.5908
0.6302
0.0000
0.8772
0.4386
0.5592
0.9346
0.7469
NPO+GD
0.6414
0.6109
0.6262
0,5465
0.6921
0.6193
0,5648
0.7668
0.6658
ME+GD
0.7271
0.9204
0.8237
0.7472
0.9313
0.8392
0.7357
0,9489
0.8423
+DO
0.7425
0.9612
0.8519
0.7316
0.9602
0.8459
0.7315
0.9625
0.8470
DPO+GD
0.7564
0.5335
0.6450
0.0000
0.8243
0.4122
0.0000
0.8041
0,4021
IDK+AP
0.7580
0.7625
0.7603
0.7529
0.7479
0.7504
0.7471
0.7433
0.7452
+DO
0.7412
0.8075
0.7743
0.7354
0.7958
0.7656
0.7362
0.7855
0.7609
DualOptim (DO) in ME+GD and IDK+AP, which perform the best in 99. to validate the effectiveness of our method. Consistent with 22 9 we consider three levels of unlearning tasks: forget 1%, 5% and 10% of the constructed data. We follow [9] and adopt the improved Model Capability (MC)
and Forger Efficacy (FE) as the evaluation metrics.
The results in Table indicate that our method significantly enhances both FE and MC in general. Although slight reduction in MC is observed for some instances, the more effective forgetting achieved ultimately leads to an increase in the average metric for all cases. Note that the standard deviations omitted since they smaller than 5% of the corresponding mean values in all cases.
In addition, as shown in Figure 7 of Appendix CC5) the unlearning process utilizing DualOptim demonstrates greater effectiveness and stability compared wtih other baselines. These findings underscore the effectiveness of our method in terms of both performance and stability for LLMs.
It is important to note that the forgetting and retaining objectives are are jointly optimized in the baselines listed in Table whereas our method alternates between optimizing these two objectives. To ensure a fair comparison, also evaluate unlearning performance by alternately optimizing the forgetting and retaining objectives using a shared optimizer. The results in Table 15 of Appendix C.5 demonstrate that DualOptim surpasses both the joint and alternate update schemes.
Table 5: Running time and memory usage of Adam and DualOptim on Llama 2 with full nall-parameter tuning and LoRA. Note that for full-parameter tuning, used two H20 GPUs and DeepSpeed zero stage 2 is adopted: for LoRA, we used a single H20 GPU.
(a) Running Time Usage (min)
(b) Memory Usage (GB/GPU)
Tuning method
MU method
Adam
DualOptim
Ratio
Tuning method
MU method
Adum
DualOptim
Ratio
ME+GD
19.40
28.50
1.47
IDK+AP
59.83
86.40
1.44
Full-purameter
IDK+AP
22.77
33.92
1.49
Full-parameter
ME+GD
59.84
86.52
1.45
LoRA
ME+GD
17.04
17.43
1.02
LoRA
IDK+AP
30.63
30.62
1.00
IDK+AP
26.76
62.32
1.21
ME+GD
31.44
30.62
0.97
Although DualOptim exhibits effectiveness in MU for LLMs, as shown in Table 5 the decoupled momentum introduces approximately 1.5x computational and memory overhead because of large amount of parameters in LLMs. To investigate our method's effectiveness under limited computation
Modeel Capability (MC) is referred to Model Utility (MU) in [9, which conflicts with the abbreviation of Machine Unlearning (MU) in this paper.

---

resources, we apply DualOptim to the parameter-effiient fine-tuning techniques, such as LoRA [54] As indicated in T of Appendix C.3] DualOptim achieves minimal compromise in unlearning performance by using L, while significantly reducing memory consumption on optimizer states.
4.4 Ablation Studies
In this subsection, we compare different combinations of optimizers to further analyze the proposed method. More ablation studies can be found in Appendix C.6
We compare various combinations of the Adam and SGD optimizers for the forget loss and the retain loss with their standalone counterparts. Besides Adam, we also investigate the combinations of SGD and other optimizers, e.g., Lion (55) and Muon [56]. The results in Table 6 indicate that the configuration of Adam (F) + SGD (R) offers the optimal balance between performance and stability. This finding underscores the importance of decoupled momentum and adaptive learning rate for the forgetting objective. Moreover, the results presented in Table 18 and Table T9 in Appendix (C66 indicate that the best optimizer for retaining depends on the choice of optimizer during pretraining and the specific MU algorithms employed. It should be pointed out that shared SGD demonstrates a competitive average performance but suffers from high standard deviation indicating instability. Conversely, Lion exhibits low variability but does not achieve optimal performance. These insights further emphasize the efficacy of our method in enhancing performance and stability in MU tasks.
Table 6: Ablation study on different optimizers when we use DualOptim SFRon. Results are based on 10% random subset unlearning task on CIFAR-10 using ResNet- 18 pre-trained by SGD. (F) and (R) denotes that the optimizer used minimize the forget and retain losses, respectively.
Optimizer
FA
RA
MIA
Gap |
Std 1
Single SGD
94.67=3.08 (0.06)
99.83 (0.17)
93,98 (0.27)
77.80=5.a1 (1.54)
0.51
2.33
Single Adam
24.54_2.1 (0.07)
99 99.9620.02 (0.04)
94.15 (0.10)
81,46_2.42 (5.20)
1.35
1.29
SGD (F) + SGD (R)
94.07=3.4 (0.54)
99.9020.06
(0.10)
93.93_0.6 (0.32)
78.48 (2.22)
0.80
2.05
SGD (T) + Adam (R)
94,58-3.40 (0.03)
99.3840.78
(0.62)
92,84=142 (0.14)
81.13
(4.87)
1.73
2.62
Adam (F) + Adam (R)
94.29
(0.32)
99 20.94 (0.06)
94.02=a0.11 (0.23)
77.86=1.50 (1.60)
0.55
0.63
Adam (F) + SGD (R)
14.69 (0.02)
99.922..0: (0.08)
94.11 (0.14)
77.77 (1.51)
0.44
0.66
SGD (F) + Lion (R)
97.88 (3.27)
99.81 2o (0.19)
93.86gu.9 (0.39)
87 87.62 (11.36)
3.80
1.36
Lion (F) + Lion (R)
94.54_1.50 (0.07)
99 20.93
(0.07)
93 (0.34)
83 83.08=1.55 (6.82)
1.82
0.81
Lion (F) + SGD (R)
94.66_1.48 (0.05)
99.91 20.03 (0.09)
93.95 (0.30)
79.55 (3:29)
0.93
0.80
SGD (F) + Muon (R)
95.74_0..4 (1.13)
99.61
(0.39)
94.04_g.0u (0.21)
84.03 (7,77)
2.37
0.32
Muon (F)+ + Muon (R)
95.12.0.54 (0.51)
99.6 99.6420.01 (0.36)
93.84_0.11 (0.41)
82.77 (6.51)
1.94
0.31
Muon (F) + SGD (R)
94.57 (0.04)
99.93 (0.07)
94.13 (0.12)
72.46_1.01 (3.80)
1.01
0.57
5
Conclusion
This study improves efficacy and stability in approximate machine unlearning (MU) methods with Dual Optimizers (DualOptim). By integrating adaptive learning rates and decoupled momentum, DualOptim enhances unlearning performance across diverse applications in computer vision and natural language processing. Its ald--and-play design ensures easy integration into existing MU frameworks. DualOptim marks a significant advancement in the MU field, offering effective solution to meet the demand for trustworthy machine learning systems.
Broader Impacts and Limitations
Our method contributes to the trustworthiness of deep learning and can be broadly applicable across diverse tasks. Although DualOptim can be integrated with parameter-efficien fine-tuning methods, it still introduces double memory consumption for optimizer states. We leave the development of more efficient approach as future work.
Acknowledgments and Disclosure of Funding
This work supported by National Natural Science Foundation of China (NSFC Project No. 62306250) and City University of Hong Kong (CityU Project No. 9220132).
10

---

References
[1] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer. Red-teaming the stable diffusion safety filter. In NeurlPS ML Safety Workshop, 2022.
[2] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. Tofu: task of fictitious unlearning for 1lms. In First Conference on Language Modeling, 2024.
[3] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning, pages 1832-884.. PMLR, 2020.
[4] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference Artificial Intelligence and Statistics, pages 1008-2016 PMLR, 2021.
[5] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1 11516-11522 2021.
[6] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei. Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations, 2024.
[7] Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient- machine unlearning with remain geometry enhancement. Advances in Neural Information Processing Systems, 37:26377-2644 2024.
[8] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems, 36:1957-1987, 2023.
[9] Xiaojian Yuan, Tianyu Pang. Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. In International Conference on Learning Representations, 2015.
[10] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[11] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[12] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141-159. IEEE, 2021.
[13] Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape of machine unlearning: A comprehensive survey and taxonomy. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[14] Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024.
[15] Seth Neel, Aaron Roth, and Saeed Sharifi- Maaiif--allvjjedi.. Descent-to-deledt Gradient based methods for machine unlearning. In Algorithmic Learning Theory, pages 931-962, PMLR, 2021.
[16] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34: 1075--888888 2021.
[17] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances neural information processing systems, 32, 2019.
[18] Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126-4142, PMLR, 2021.
11

---

[19] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[20] Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. A swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1139-1147. PMLR, 2019.
[21] Anvith Thudi, Gabriel Deza, Varun Chandrasckaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303-319. IEEE, 2022.
[22] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3304-9312, 2020.
[23] Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[24] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584-51660 5065,, 2023.
[25] Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 37, pages 7210-7217, 2023.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[27] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[29] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFE/CVF international conference on computer vision, pages 4195 2023.
[30] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22522-22531, 2023.
[31] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito. and Eric Wallace. Extracting training data from diffusion models. 2023.
In 32nd USENIX Security Symposium (USENIX Security 23), pages
[32] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2426-2436, 2023.
[33] Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22691-22702, 2023.
[34] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me- not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1755-1764, 2024.
[35] Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36: 16:7717111799 2023.
12

---

[36] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09280 2023.
[38] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038-2047, 2022.
[39] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023.
[40] Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations, 2024.
[41] Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. Avoiding copyright infringement via machine unlearning. arXiv e-prints, pages arXiv-2406, 2024.
[42] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pages 1-14, 2025.
[43] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024.
[44] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2615 2632, 2021.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[47] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the JEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[48) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 15:211-252, 2015.
[49] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[50] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. Annual Network and Distributed System Security Symposium, 2023.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234-241. Springer, 2015.
[52] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
13

---

[53] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.0544 00..04639 2023.
[54] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[55] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances neural information processing systems, 36:49205-49233, 2023.
[56] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
14

---

A Proofs
A.1 Proof of Lemma 3.4
Proof. Let 0,,02 02 be independent copies of with probability density p(). Using the variance
identity Var(X) = [ X2]l²] for independent copies X1. X2, we have:
1 Var(VoC(()) - //2 // p(0)))(00)) (V V
By L-smoothness, IVV((6)) - VC( - 4110 - -221, we have: L2
Var(VoL(0)) s 2 /| p(01))((22)1) -
(8)
Since Var(0) < 0g we have 2 p(0)((1))00))000 02112d0,d02 S Sg.. Therefore, we can conclude Var(V&L(()) - o
Discussion. Lemma 3.4 is important tool for analyze the variance of the full-batch gradient, because the full-batch gradient is calculated based on the model parameters obtained in the last iteration, which is also random variable. Specifically, the update rule in (5) for further analysis. We add no superscripts, indicating it is applicable for both update schemes.
derive the following inequality from
Var(g 2 0² Var(Vo s + (a
Var(Dr.)) S 00² Var( Var(VoL.(00..) S +LVVar(0,,))
A.2 Proof of Theorem 3.5
Proof. Without the loss of generality, we can assume the pretrained parameter 0, = and the learning
rate 1) = 1, because shift and multiplication do not change the inequalities in the conclusion to prove.
Unfold the update rules in (5) and we obtain
m S
S am
r,t + 9/.1 Ei a -i 2g
1=0 a g Zizo
(Shared Momentum)
m om
+ gh, Ei 22-00+
(1)))))
ftt
m f.t am
D
f (-1 + P Ei= a 1 "
D
(Decoupled Momentum)
m,,t
D
am
D r1-1 + I Ei
0i= a=
--i gg.
(10)
For notation simplicity, we let A Σk a'. It clear that bkg S K2, As I Aka-
Based on the update scheme and 17 1, we have:
85 Σ1
=0 m
+ 1=0
m F.j
(Shared Momentum)
05 Σi 0 m
f.i
+
Нi i=0 n..
!
(11)
(Decoupled Momentum)
00 = Ei=
m
D
+
i=0 m D
r.a
0 = Di m 7
-
D
+
Σ Di=o m
Combining 110) and 1I we have the following equations. For notation simplicity, we let A
Σtert at. is clear that Vk S k2, Ar, S Akg.
0
ft =1=0 A20--199 + Σ/=1 A2(1-1)-195
(Shared Momentum)
2g + E 88 = Σi A2
== A A2
(12)
00 Σt
A + Σ 0 A
(Decoupled Momentum)
0 = Ei=
A + E A A
15

---

We prove the theorem by mathematical induction.
We start with the of O. Based 12 we have 07.0 -7700 f.
--
-
Both
stochastic gradients calculated on the initial parameter Co. so based on Assumption
3.1
we have:
Var (07,0) S 0² del (,.)
Var(0 1.0) S 5 del Var(Of.)).
(13)
In addition, we have 0 -j.00 gr.0 and 0r = -(1 (1 + +0)g7, 9..0 based on 12
Based on
Lemma 3.4 and inequality 9 we have:
Var(Or)) S 0² + Var S +10? Var(ā S o² -ara(fffo) s (L2 1)o²
(14)
We assume negative correlation between gradients from the forget loss and the retain loss in Assump- tion 3.2 Therefore, can derive the variance for 8700 and 6 as follows:
Var(Os.) 7.0) S (1 + a1+)))aar(@pp) + Var(P S (1++a) + (T. + 1102 del Var(0,,))
(15)
Var(o I Var( + Var(P S 0² + +1+ (1+ # Var(0..).
Thus, have the following conclusion:
Var Var(OP.o) S <Var(e),))
Var(e (PP))) I &Va((e (OS,))
(16)
That is to say, the conclusion of the theorem holds for t 1= 0. Now we assume that the conclusion of the theorem holds for 0, I..., t - 1, and then consider the case of
Based on Assumption 3.2
we have the following inequality for both update schemes:
Ywo.... ..., We. Var
Σ w
Σ w Var (g.( +
+22 Σ
(0.)),),, v Var(@r.)Var((r..)
ff
(ar(
+
Σ
Var + w = (1+
Σ
W
2
Var (ār(
1=0
$
1=0
(17)
Similarly, we have Vwo, Wt, Var EE wiGr.) S (1 E1+ Var((nn).)
Now we combine 12)
14).
and can derive the following inequalities:
5 (1 5 (1
Σ A/ (d² 11 (2 (1+ (t- 1)7) Σ
Agi -)- (2+ ++?Var(00)))) Ag
1=0
1=0
1-1
Var(Of,)) 5 (1+ Σ
Al.s (o² ((V +1+ (t 1)T)
Σ
AZ (o* L'Vrr((()
k=0
1=0
Var(o 5 (1 Σ
12 A2 +2+12' ( (1 (T)
Σ
A2 (0² + (arr)((0).)))
i=0
1=0
Var(e S (1+ Σ Al_ (0"+L'Varro + (1+ Σ Al (0" L'Var((PP)))
(18)
By definition, Var Var(O j.c), (ar().)), and Var(O are the maximum possible value of
Var(0?,). Var(o ) Var(oP.) ) and Var(oS respectively. Considering the inequalities in [18)
all achievable, we have the following:
16

---

-ar(6),))(11+5))
- (1+tT) Σ 1) (o' L'V + (1+ (t (t-1))
Σ
1 1) (o* L
Var(6f))=(1+000))
= (1+ Σ Ai-i (o" (L (1+ (t 1)r)
Ah-i-1 0
-
= (1+ A +) (o* L2 (1+ Σ Ag(t-o) (o² + L Var(())))
Var(O,.) = -1 (1+ Σ A L L'Var(0 (1+ Σ Ap (o² + Var((()).))
1=0
(19)
By induction, we have for t' {0, 1, 1,..,t - 1}, Var(O Var(0 f.e)) Var
Var
In addition, yk; < 122 Ag, I Akg Therefore, after comparing the factors and terms the right hand of (19), it is obvious to obtain the following conclusion.
Var(Of.)) S Var(O f.t)s
Var(O
I (.
(20)
By induction, the conclusion of this theorem holds for any f 2 0.
A.3 Variance Bound of Performance Metric Function
We let (0) to represent the performance of model with parameter 0 where M can be FA/RA/TA/MIA. Before we derive the variance bound of performance metric function (0), we make the following assumption:
Assumption A.1. The performance metric function M is Lipschitz continuous:
101, 62, [M(()) - M(22)) S Las 0211,
(21)
where LM is nen-nvgative constant.
Following a similar logic to Lemma 3.4 we have the following corollary:
Corollary A.2. If Assumption A.1 holds, and the parameter variance satisfies Var(O) Var(O), then have Var(M <. L3, Var(O).
Proof. Similar to the proof Lemma 3.4 we have: 1 Var(M(()) = 2 /f p(0))p(02) [M())) - M(22))l d0,d02,
(22)
where 01, 02 are independent copies of with probability density p(0).
By (01) M (02)ll < LM 02/1. we have:
Var(M(()) < L /f //00)100>>>00 -0112d0,,00 d00.
(23)
Since Var(O) < Var(O), we have ʃf (002))111 - 0211=d0,d02 S Var(O). Therefore, we can conclude Var(M < L2 Var(O).
Discussion. The final unlearned parameters are usually the parameter updated by retaining loss at the last iteration T, i.e., P.r using decoupled momentum and Sh using shared momentum. According Let the upper bound the performance variance (8)( L3, Var(O), we derive Var(M < (ar(M Thus, decoupled momentum can also induce less variability on performance metrics.
to Theorem 3.5 we have (ar(OPT) & Var(o
17

---

B Implementation Details
B.1 Implementation Details for Image Classification
For CIFAR-10. CIFAR-100, and SVHN using ResNet-18, all baselines use the SGD optimizer with momentum of 0.9, weight decay of 5 x 10 and batch size of 128 if not specified. For TinyImageNet, Swin-T, the models are initialized from torchvision weight pre-trained on ImageNet. All baselines use the Adam optimizer with weight decay of 5 x 10 and batch size of 128 if not specified. Summaries of the hyperparameters for each method on each dataset are shown in Table T0 Note that the unspecified hyperparameters are the same as the default ones reported in their original papers.
Table 7: Summary of hyperparameters for each method on unlearning 10% random subset of CIFAR- 10. 7) is short for learning rate.
Method
Hyperparameters
Pretrain RT
epoch = 200, cosine scheduler, y) = 0.1
epoch 200, cosine scheduler, 1) 0.1
FT
epoch 10, constant scheduler, 0.01
GA
epoch 10, constant scheduler, "
10-4
RI SCRUB
epoch 10, constant scheduler, "y 0.01, 7, 0.01
epoch 10, constant scheduler. Adam, /// 1 10 .q. 1 10
SalUn
epoch 10, constant scheduler, "/ 0.018, frr 0.01
SFRon
Tout 1500, T.. 5, cosine scheduler, f/ 0.01,
0.01, =
SCRUB+DO
epoch 10, constant scheduler, Adam (F) + Adam (R), M/ - 1.5 x 10
1 10-4
SalUn+DO
epoch = 10, constant scheduler, Adam (F) + SGD (R), 77 = 1.3 10
. 0.01
SFRon+DO
Tout = 1500, T.. = 5, cosine scheduler, Adam (F) + SGD (R). = 10 0.01, ==
Table 8: Summary of hyperparameters for each method on unlearning 50% random subset of CIFAR- 10. 17 is short for learning rate.
Method
Hyperparameters
Pretrain RT
epoch =
200, cosine scheduler, = 0.1
epoch =
200, cosine scheduler, = 0.1
FT'
epoch 10, constant scheduler, 1 0.01
GA
epoch 10,
scheduler.
10-6
RL SCRUB
epoch 10, constant scheduler, fy 0.01, 7/- 0.01
epoch 10, constant scheduler, Adam, n1 1.6 10-3, 10-4
SalUn
epoch 10, constant scheduler, N/5 2.5 10
SFRon
T 1500, T.. cosine scheduler, f/ 0.011 00.1,
SCRUB+DO
epoch = 10, constant scheduler, Adam (F) Adam (R), = 2.8 10 =1 10
SalUn+DO
epoch = 10, constant scheduler, Adam (F) + SGD (R). 5.3 x 10 00-5500--00000000000000000 0.01
SFRon+DO
Tout
1500, Tin J, cosine scheduler, Adam (F) SGD (R), 7/ = 4.1 x 10 1.' 0.01, a=
Table 9; Summary of hyperparameters for each method on unlearning 10% random subset of CIFAR- 100. is short for learning rate.
Method
Hyperparameters
Pretrain RT
epoch 200, cosine scheduler, 0.1
epoch = 200, cosine scheduler, 1) 0.1
FT
epoch = 10, constant scheduler, n = 4.5 10-2
GA
epoch = 10, constant scheduler, n 5.2 x 10
RL
epoch = 10, constant scheduler. = x 10-1 = 0.01
SCRUB
epoch = 10, constant scheduler, Adam, // = 6 10-5, 10-5.%=1111 =1 10
SalUn
epoch = 10, constant scheduler, = 1.3 x 10-3, 1mn-000 0.01
SFRon
Tou: = 1500, T.. = 5, cosine scheduler, 7, 0.01, =1===000000 =.01,
SCRUB+DO
epoch 10, constant scheduler, Adam (F) + Adam (R). fl 1 10
10
SalUn+DO
epoch = 10, constant scheduler, Adam (F) + SGD (R), 1y = 1.9 10 " 0.01
SFRon+DO
Tixt 1500, T., 5, cosine scheduler, Adam (F) + SGD (R), 7/ = 10 0.01,
18

---

Table 10: Summary of hyperparameters for each method on unlearning 10% random subset of SVHN. is short for learning rate.
Method
Hyperparameters
Pretrain RT
epoch =
200, cosine scheduler, = 0.1
epoch =
200, cosine scheduler. = 0.1
FT
epoch 10, constant scheduler,
10
GA
epoch 10, constant scheduler, = x 10-4
RI
epoch 10, constant scheduler, // 8 10-3, () 0.01
SCRUB
epoch = 10, constant scheduler, Adam, /// =8x 10
x 10
SalUn
epoch = 10, constant scheduler, /// 1.25 x 10 .V,, 0.01
SFRon
Tour 1500,, T.n cosine scheduler, f/ 0.01, 7r =0011 12.5
SCRUB+DO
epoch = 10, constant scheduler, Adam (F) Adam (R),
2.2 10 = 1 10-4
SalUn+DO
epoch = 10, constant scheduler, Adam (F) + SGD (R). 2.5 10 =.01
SFRon+DO
T
1500, Ti, = 5, cosine scheduler, Adam (F) SGD (R), f// 1.6x 10 = 0.01, a=
Table 11: Summary of hyperparameters for each method on unlearning 10% random subset of Taaee v is short for learning rate.
Method
Hyperparameters
Pretrain RT
epoch 10, cosine scheduler, y = 10
epuch 10, cosine scheduler, x 10
FT
epoch 1. constant scheduler. 1
10-4
GA
epoch 1, constant scheduler, 17
10-6
RL
epoch I. constant scheduler. /// 5 10-5. 000, =1 10-10 10-1
SCRUB
epoch constant scheduler, Adam, fJ = 10 2 10-5
SalUn
epoch 1, constant schoduler. N/ 9 10-5, 0000-.0--99100 10-5
SFRon
Tout 500, Thp 1, cosine scheduler, fy = 3 10
= 10 100
SCRUB+DO
epoch 1. constant scheduler, Adam (F) Adam (R), -2 10 ...- -2x10
SalUn=DO
epoch 1, constant scheduler, Adam (F) Adam (R), 9 1000, =9x
SFRon+DO
Tout 500, Tm = 2, cosine scheduler, Adam (F) + Adam (R), = 1.9 10 11
B.2 Implementation Details for Image Generation
For CIFAR-10, we use DDPM based on U-Net architecture with 1000 timesteps for linear schedule.
All methods use Adam optimizer and batch size of 128. The hyperparameters of SalUn+DO are:
10 10-5 7r = 10-1, threshold = top-50%. The hyperparameters of SFRon+DO are: Tout 150, Tin ny 1 10 4, 17.
Tout 150, T., = 1, f)
10
1, X 0.5, 7 3.
The hyperparameters of other methods are the same as those reported in [7].
For ImageNet, optimizer and batch size of 1. The hyperparameters of SFRon+DO are: Tout = 500, Tin = 1, T = 5 x 10-5, n 5 10 1, = 3. Since the batch size is 1, we ignore À in adaptive coefficients. The hyperparameters of other methods are the same as those reported in (7].
pre-trained DIT-XL/22 with 256 x 256 resolution. All methods use AdamW
B.3 Implementation Details for Natural Language Processing
For Phi-1.5 and LLaMA 2, we utilize pre-trained models on the TOFU dataset (2] and conduct evaluations accordingly. For the baseline methods, including GA+GD and DPO+GD, well as the methods ME+GD and IDK+AP, we adopt the default hyperparameters reported in 19. For IDK+AP, the forget coefficient and regularization coefficient are set 1.0 across all models. For ME+GD, the retain coefficient is 1.0 for all methods. The forget coefficient is 1.0 for LLaMA 2, while it is set to 0.1 for LLaMA LoRA and Phi 1.5. For the LLaMA LoRA configuration, the LoRA rank is and the LoRA alpha is 32, For training LLaMA 2 with DualOptim, we use two NVIDIA H20 GPUs with 96GB of memory each. For Phi-1.5, we use two NVIDIA RTX 6000 Ada GPUs with 48GB of memory each. The detailed hyperparameters of our methods are reported in Table 12
19

---

Table 12: Summary of hyperparameters for each method LLM unlearning task. of is short for
learning rate.
Model
Method
Hyperparameters
ME+GD+DO
epoch 5. 1/r 10 77. x 10
Phi-1.5
IDK+AP&DO
epoch = 5. Vir =
1/- 10
ME+GD+DO IDK+AP+DO
epoch = 5, = 1.45 x 10 1/- 10
LLaMA
epoch 5, // 1.25 10 ,/- x 10
ME+GD+DO IDK+AP4DO
epoch = 5, f/ = x
" = 22 10
LLaMA LoRA
epoch = 5, 175 = 5 10
TT- = 2
C
Additional Results
C.1
Non-positive Correlation between Forget and Retain Gradients
Figure 4 iillustrates non-posinne non-positive correlation between 9/ and 9. during unlearning, supporting the reasonableness of As- sumption 3.2 Note that the results are obtained using SGD optimizer.
0.02
co
-0.04
-0.20
C.2 Unlearning Process of Other MU Methods
-0.08
The unlearning processes of SalUn [6 and SCRUB g are illustrated in Figure 5 and respectively. Compared SFRon (see in Figure 2). SalUn and SCRUB exhibits less performance fluctuation. However, they underperform SFRon by a large margin. Despite that, our method is still effective in boosting their unlearning performance.
620
Iteraioo
Cosine similarity be- tween Øj and 9r- The moving age curve is shown for better visu- alization.
Figure 4:
Ade
SCD
912
Dasoptre
115
Itention
(a) Forget Accuracy (FA) (b) Retain Accuracy (RA) (c) Test Accuracy (TA)
(d) MIA
Figure 5: Unlearning process of SalUn. All results are obtained from unlearning 10% random subset of CIFAR-10 10 on ResNet-18.
S-B
Ader
17.
Adm
VOD
5
8172
61C
44m
800
3550
banzznn
Renittnn
Brnidon
(a) Forget Accuracy (FA) (b) Retain Accuracy (RA) (c) Test Accuracy (TA)
(d) MIA
Figure 6: Unlearning process of SCRUB. All results obtained from unlcarning 10% random
subset of CIFAR-10 on R8.Net-18.
C.3 Additional Results of Image Classification
We further conduct experiment to evaluate the performance of unlearning 50% random subsets of CIFAR-10 10 and 10% random subsets of CIFAR- and SVHN. The results are reported in Table T3 The observation Table [3] is consistent with that Table emphasizing the efficacy of our method across different fractions of forget data, different datasets and different model architectures.
20

---

Table 13: Additional performance summary of MU methods for image classification. (a) 50% random subset of CIFAR-10, (b) 10% random subset of CIFAR-100 and (c) 10% random subset of SVHN. ResNet- 18 is used. All results are presented mean and standard deviation across 5 trials with different random data. Performance gap from RT are indicated in blue.
CIFAR-10 Random Subset Unlearning (50%)
Method
UA
RA
TA
MIA
Gap
Std 4
Retrain
93,3610:10 (0.00)
10000000.00 (0.00)
93.11 20:10 (0.00)
69.02_0.or (0.00)
0.00
0.10
FT
99.28...00 (5.92)
99.92. 0n (0,08)
94.20
(1.09)
88.77 (19.75)
6.71
0.14
GA
95,44 (2.08)
95.53 30 (4,47)
90.54 (2.57)
89.14 sn (20.12)
7.31
7.03
RL
99.2610.00 (5,90)
99.50 (0,50)
93.91
(0.80)
58.50 (10.52)
4.43
0.36
SCRUB
97.07 (3.71)
99.57 (0.43)
93.21 0 (0.10)
80.62:1.14 (11.60)
3.96
0.42
DualOptim
97.42.10.09 (4,06)
99.52 05 (0.48)
93.38 20 (0.27)
77.94_0 (8.92)
3,43
0.22
SalUn
99.05
(5.69)
99.88 2880..0 (0.12)
94. 94.10 10 (0.99)
68. 10.30 (0.72)
1.88
0.24
DualOptim
93.82.+0.66 (0.46)
97. 77,902o.a s6 (2.10)
90.66 (2.45)
68.88 04 (0.14)
1.29
0.40
SFRon
+2.54 (0.15)
98. 98.90 no (1.10)
91.34 22 (1.77)
71.75_3.31 (2.73)
1.44
1.87
-DualOptim
91.38+0.51 (1.98)
99.52 (0.48)
91.07 -0.31 (2.04)
69.66 0.33 (0.64)
1.29
0.41
(b) CIFAR-100 Random Subset Unlearning (10%)
Method
UA
RA
TA
MIA
Gap 4
Std 4
Retrain
77.9620.55 so (0.00)
99.98 oo (0.00)
37 (0.00)
41.79 (0.00)
0.00
0.21
FT
17.97 (0.01)
92.33 (7.65)
66.07.20.45
(11.16)
62. 62.64
(20.85)
9.92
0.31
GA
81.81 (3.85)
83.52+11.25 (16.46)
63.
(15.10)
12.62=14.47 (30.83)
16.56
11.76
RL
94.66-0.21 (16,70)
98. 98.10+0.03 (1.88)
70.16:0000 (7.07)
50.82220.51 (0.97)
6.66
0.21
SCRUB
77.4540.50 (0.51)
99.26
(0.72)
73.51.0.37 (3.72)
70.81 (29.02)
8.49
0.45
DualOplim
76.72.1.60 (1.24)
99.38 (0.60)
73.87 (3.36)
(21.27)
6.62
0.35
SalUnn
95.02-0.44 (17.06)
98.04 (1.94)
70.26 (6.97)
41.35 (0.44)
6.60
0.30
DualOptim
78.04+1.07 (0.08)
97. 97.9640.09 (2.02)
71.10=0.25 (6.13)
41.9440..0 en (0.15)
2.10
0.53
SFRon
16.24_12.35 (1.72)
99.56.1.22 25 (0.42)
73.1821.30 (4.05)
57.64 (15.85)
5.51
5.33
DualOptim
74.32+4.28 (3.64)
99.71 (0.27)
73.66 (3.57)
54.02 54.02+1.88 (12.23)
4.93
1.59
(c) SVHN Random Subset Unlearning (10%)
Method
RA
MIA
Gap
Std. J
Retrain
96.05 (0.00)
99.82.0.01 01 (0.00)
96.53 (0.00)
79.47 (0.00)
0.00
0.07
FT
99.51 (3.46)
100.00
(8.18)
95.85 +0.03 (0.68)
80.23+ 66 (0.76)
1.27
0.19
96.81.4..4 (0.76)
4. (2.41)
92.60 (3.93)
88.80 ou (9.33)
4.11
4.77
RL
94.7320.32 (1.32)
99.43 (0.39)
94.5620.24 (1.97)
74.00 (5.47)
2.29
0.42
SCRUB
92.41+0.22 (3.64)
99,82-0.08 (0.00)
94.68 11 (1.85)
04.44+ (4.97)
2.61
0.23
DualOptim
95.42=0.22 (0.63)
99.82=0.01 (0.00)
95.06 (1.47)
82.70.+0.34
(3.23)
1.33
0.19
SalUn
94.69+0.12 (1.36)
98.85 (0.97)
9..5440..a1 (1.99)
73.11+0.47
(6.36)
2.67
0.33
DualOptim
99.58 0s (3.53)
100.00 ou (0.18)
95.76.0.04 (0.77)
79.59
(0.12)
1.15
0.14
SFRon
91.29 (1.24)
100.00 (0.18)
95.41
(1.12)
79.25 (0.22)
0.69
L.11
DualOptim
98.20 (2.15)
100.00 00 (0.18)
95.6320.00 (0.90)
79.56 (0.09)
0.83
0.44
C.4 Additional Results of Image Generation
To further validate the effectiveness of the proposed method, we apply DualOptim to SalUn [ on CIFAR-10 with DDPM. Table 1 illustrats that DualOptim can still enhance the performance and stability of SalUn in image generation tasks. Note that we adapt SalUn to alternate updating scheme when applying DualOptim, which originally utilizes joint updating scheme in image generation tasks. We do not include its results on ImageNet due to suboptimal and unstable performance.
Table 14: Class-wise unlearning performance of SalUn+DO CIFAR-10 with DDPM. The best unlearning performance for each forgetting class is highlighted bold for FA (in %) and FID.
Automobile
Cat
Dog
Horse
Truck
Average FA
Method
F FID 1 FA
FID
4. FA L
FID
FA 4 FID 4. FA 4 FID 4
FID
SalUn
0.20
21.23 1.40
20.29
0.00
20.18
0.60 20.70 0.80 20.45 0.60+0.40
20.57
+DO
0.00 19.93 0.00 20.45
0.00
20.12
0.00
20.14 0.00 19.91 0.00
20.11
21

---

C,5 Additional Results of Large Language Models
The results in Table [5| suggest that the alternate updating method can improve LLM unlearning. In addition, DualOptim boosts the performance based on that. As presented in Table 16 DualOptim achieves minimal compromise in unlearning performance by using LoRA, while significantly reducing memory consumption. Furthermore, Figure 7 illustrates that the unlearning process utilizing DualOptim demonstrates greater effectiveness and stability compared to other baselines.
Table 15: Performance comparison of different updating methods on TOFU-finetuned Phi-1.5. The MU method is IDK+ The results include Model Capability (MC), Forget Efficacy (FE), and the average metric (Avg.) for forgetting 1%, 5%, and 10% data. Note that Joint represents jointly optimizing forgetting and retaining objectives, which is the default updating method; Alternate represents alternately optimizing these two objectives using a shared optimizer.
forget 1% data
forget 5% data
forget 10% data
Method
MC f
FE f
Avg.
MC f
FE1
Avg. f
MCT
FE f Avg. 1
Joint
0.4403
0.5723
0.5063
0.4800
0.5112
0.4956
0.4614
0.6003
0,5308
Alternate
0,4182
0.5746
0.4964
0,4348
0.6570
0.5459
0.4588
0.6619
0,5603
DualOptim
0.4221
0.7037
0.5629
0.4633
0.6974
0.5804
0,4422
0.7193
0.5807
Table 16: Performance comparison of different MU methods TOFU-fietuned LLaMA 2 with LoRA. We set the LoRA rank to 8 and the LoRA alpha to 32. The results include Model Capability (MC), Forget Efficacy (FE), and the average metrie (Avg.) for forgetting 1 1%, 5 5%, and 10% data.
forget % data
forget 5 % data
forget 10% data
Method
MC f
FE f
Avg. f
MC f
FE
Avg. t
MC1
FE f
Avg.
GA+GD
0.5007
0.6051
0,5529
0.5470
0.4306
0.4888
0,5745
0.9133
0.7439
NPO+GD
0.5290
0.5778
0.5534
0,5185
0.7032
0.6109
0.5350
0.7745
0.6548
ME+GD
0.7526
0.8425
0.7976
0.7435
0.9298
0.8367
0.7410
0.8856
0.8133
+DO
0.7542
0.9646
0.8594
0.7373
0.9545
0.8459
0.7363
0.9549
0.8456
DPO+GD
0.6874
0.7647
0.7260
0.6951
0.5490
0.6221
0.7308
0.3973
0.5640
IDK+AP
0.7572
0.6754
0.7163
0.7471
0.7430
0.7451
0.7604
0.7411
0.7507
+DO
0.7422
0.7729
0.7575
0.7311
0.7499
0.7406
0.7533
0.7532
0.7533
C.6
Additional Ablation Studies
Decouple m and in Adam. Given that Adam incorporates two momentum terms, i.e., the first moment m and the second moment v. we conducted a further analysis to explore the impact of decoupling these terms. As shown in Table [7] decoupling both m and yields the best performance. This finding reinforces the importance of fully decoupling the optimization processes for forgetting and retaining objectives.
Table 17: Ablation study on Dual Adams. Results are based on 10% random subset unlearning task on CIFAR-10 10 using ResNet-18 pre-trained by SGD. SFRon is the adopted MU algorithm. and denote the decoupled first and second moment terms in Adam, respectively.
m 17
BA
RA
TA
MIA
Gap 1
Sid
x
24 (0.07)
99.96 02 (0.04)
20 (0.10)
81.4622.42 (5.20)
1.35
1.29
x
94.61 a2 aa (0.00)
99.95 1 (0.05)
94.26
(0.01)
82.44 (6.18)
1.56
1.33
94.52+2.44 44 (0.09)
99.94 08 (0.06)
94.09 (0.16)
82,06 (5.80)
1.53
1.45
V 1. (0.32)
99.94+0.001 (0.06)
94.02+0.111 (0.23)
77.86+1.30 (1.60)
0.55
0.63
Ablation studies on Adam-trained ResNet-18 and SCRUB (8). We conduct ablation studies based ResNet- 18 + SCRUB. As illustrated in Table [8 and 19. Adam (F) + Adam (R) is the best configuration in both cases, indicating that the optimal optimizer for retaining is influenced by the choice of optimizer during pretraining and the specific MU algorithms employed.
on the configurations of Adam-trained ResNet-18 + SFRon and
Shared optimizer with larger momentum coefficient. We further compare DualOptim with shared optimizers that employ larger momentum coefficients. As observed in Table 20 while increasing
22

---

GA+GD
ME+GD
ME+GD+DO
ferget
forget 3%
forget
0.8
Forget Efficacy
0.7
0.6
0.4
0.3
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Capabilty
(a) untargeted unlearning
IDK+AP+DO
Torget 1% data
forget 5% data
Torget 10% data
Forget Effiracy
0.7
0.6
0.4
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Model Capabilly
(b) targeted unlearning
Figure 7: Forget Efficacy versus Model Capability of (a) untargeted unlcarning and (b) targeted unlearning on TOFU with Phi-1.5. The relative size of the markers indicates the epoch of unlearning.
Table 18: Ablation study on different combinations of DualOptim. Results are based on 10% random subset unlearning task CIFAR-I0 using ResNet-11 pre-trained by Adam. SFRon is the adopted MU algorithm.
(F) and (R) denotes that the optimizer used for minimizing the forget and retain losses, respectively. Note the the result of RT is reported since the pretrained model is different from that in Table 6
Optimizer
BA
MIA
Gap
Sid J
RT
93 (0.00)
100.00 (0.00)
92.84.0.00 (0.00)
78.36 (0.00)
0.00
0.30
SGD
93.88 a1 (0.44)
20 (1.31)
92.17 65 (0.67)
12
Adam
94.54 (1.10)
99.61 (0.39)
92.02 (0.82)
76.10_4.40 (2.26)
1.14
1.78
Adam Adam
93.12 29 (0.32)
99.33 oa (0.67)
91.6410-21 (1.20)
78.63 8. (0:27)
0.62
0.60
Adam (F) + SGD (R)
95.30 as (1.86)
99.45 (0.55)
92.6520.18 (0.19)
18.13 (0.23)
0.71
0.37
the momentum coefficient a can reduce performance variation when using SGD, it also results in slower convergence and ultimately leads to suboptimal performance. In contrast, increasing 31 has only a marginal impact when using Adam. This is because Adam's 's momentum update rule, i.e. m 3 m (1 81 )9, inherently provides stronger smoothing effect compared to SGD, i.e. m = amm when the same momentum coefficient is applied.
C.7 Visualization for Machine Unlearning in Image Generation
Generated images from the unlearned model utilizing DualOptim are shown in Figure and The visualization indicates that, by leveraging DualOptim, effective unlearning is achieved and the generation capability for the remaining classes is retained.
23

---

Table 19: Ablation study on different combinations of DualOptim. Results are based on 10% random subset unlearning task on CIFAR-I0 using ResNet-18 18 pre-trained by SGD. SCRUB is the adopted MU algorithm. (F) and (R) denotes that the optimizer is used for minimizing the forget and retain losses, respectively.
Optimizer
FA
RA
TA
MIA
Gap
Std
SGD Adam
94.64_0.28 0.25 (0.03)
99 10 (0.56)
93 22 (0.76)
80.37 so (4.11)
1.37
0.36
92.88 (1.73)
90 (0.38)
93.54+0.22 (0.71)
82.78+0.86 (6.52)
2.33
0.36
Adam (F) + Adam (R)
94,9020.42
(0.29)
99.5220.00 (0:48)
93.50 (0:75)
78.26 T9 (2.00)
0.88
0.38
Adam (F) + SGD (R)
g4.20 (0.41)
98.68g0.17 17 (1.32)
92.58 (1.67)
78.28 12 (2.02)
1.36
0.53
Table 20: Comparison between DualOptim and shared optimizers with larger momentum coefficients. Results are based on 10% random subset unlearning task CIFAR-10 using ResNet- 18 pre-trained by SGD. SFRon is the adopted MU algorithm. Note that 31 is the momentum coefficient in Adam
Optimizer
RA
Cap
SGD to 0.9)
94.67+3.0s (9.06)
99. 99.83++..15 (0.17)
93.9 93.98+0.556
(0.27)
77.80+5.01 (1.54)
0.51
2.33
SGD 0.95)
94.44 (0.17)
99.82 22 (0.18)
94.1110.55 (0.14)
38.43 (2.17)
0.67
1.32
SGD (a 0.99)
95.0610.04 (0.45)
99.02 (0.98)
93.63 (0.62)
78.6911.13 (2.43)
1.12
0.53
Adam (,1 0.9)
94.54:2.41 (0.07)
99.96. (0.04)
94.15 (0.10)
81.46:2.:2 (5.20)
1.35
1.29
Adam (91 0.90)
94.64+2.61 (0.03)
99.93+0.02 (0.07)
94. 94.1220.35 (0.13)
82. 1042.40 (5.84)
1.52
1.32
Adam (3, 0.999)
2. (0:05)
99.79 os (0.21)
93.75 20.58 (0.50)
80.3632..b (4.10)
1.22
1.46
DualOptim
94.69+1.18 1.13 (0.02)
99.92+0.01 (0.08)
94.11+0.11 (0.14)
1. (1.51)
0.44
0.66
(g) Forgetting Frog
(h) Forgetting Horse
(i) Forgetting Ship
(i) Forgetting Truck
Figure 8: Visualization of class-wist unlearning results classifier free guidance DDPM on CIFAR-10 10. The forgetting class is marked with red color.
24

---

(a) Forgetting Airplane
(b) Forgetting Automobile
(c) Forgetting Bird
(d) Forgetting Cat
(c) Forgetting Deer
(l) Forgetting Dog
25

---

(a) Forgetting Cockatoo
(b) Forgetting Golden Retriever
(c) Forgetting White Wolf
(d) Forgetting Arctic Fox
(c) Forgetting Otter
Figure 9: Visualization of class-wise unlearning results on DiT on ImageNet. The forgetting class is marked with a red color.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

For each theoretical result, the paper provides the full set of assumptions and complete proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
This paper discloses all the information needed for reproduction.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
This paper provides open access to the code in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides/ Stwriidde/CCoeeuubbsssstteee Policy)l for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7
htte//CoooSSSiiieeeeee wwweiisssoonoooni ooic for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper report the standard deviation of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This paper provides sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwrr..cppppaaococccccaaa
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses broader impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper utilizes public pretrained language models and image generators and poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets, used in the paper, are properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, thcoe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
32

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettps://neuripee cwwparr.pp.cccoooeeeeeeeeeeee for what should or should be described.
33


Reviews:
[{"review_id": "112DYDjvtB", "reviewer": "Reviewer_vno1", "summary": "This paper introduces DualOptim, a method to resolve the instability and difficult tuning associated with existing machine unlearning (MU) methods. DualOptim employs two separate optimizers: an adaptive one, like Adam, for the volatile \"forget\" objective, and the original optimizer for the \"retain\" objective to preserve model performance. By decoupling the momentum between these two tasks, the method is shown to enhance the stability and efficacy of the unlearning process. Extensive experiments validate that DualOptim boosts performance across diverse applications, including image classification, image generation, and large language models.", "questions": "1. To quantify practical overhead, please report the wall-clock time and peak memory usage for DualOptim compared to a key baseline on a representative task.\n\n2. To validate your stability claims, please detail the hyperparameter tuning budget for both DualOptim and the baselines to ensure a fair comparison.\n\n3. Please address the mismatch between your theory (proven for SGD) and your algorithm (which uses Adam) by discussing the limitations and applicability of your proof.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n\n1. The idea is clear and intuitive. Using Adam for the “forget” objective and SGD for the “retain” objective is easy to implement and generally sensible.\n\n2. The experiments are comprehensive, spanning image classification, diffusion models and LLMs, showing the approach is not task-specific.\n\n3. Table 5 systematically varies the two optimizers and supports the chosen pairing (Adam-F + SGD-R)\n\n4. Authors acknowledge memory overhead in Broader-Impacts/Limitations, showing awareness of practical constraints\n\nWeaknesses:\n\n1. The entire proof is for plain SGD with momentum (η=1 is also hard-coded)  , but DualOptim actually uses Adam on the forget branch. Preconditioning fundamentally changes the variance dynamics, so the guarantees do not directly transfer.\n\n2. The paper's theory culminates in Theorem 3.5, which proves a smaller worst-case parameter variance for decoupled momentum. However, it does not mathematically connect this reduced variance to better performance on downstream metrics like the \"Gap\" to retraining or Membership Inference Attack (MIA) success.\n\n3. The paper states that all baselines were tuned via \"sophisticated search,\" but does not detail if the search budget was equal for the baselines versus DualOptim, which introduces two separate learning rates. his potential imbalance undermines the fairness of the comparison and could be the source of the reported stability gains.\n\n4. The main results compare DualOptim's alternating update schedule with baselines that sometimes use a joint optimization objective. While a fairer comparison is included in Appendix C.5, it is not featured in the main paper. This makes the primary results potentially misleading from a practical implementation standpoint.", "was_revised": true, "final_justification": "The authors have clarified the methodology, which resolves my primary concern about the experimental setup.\n\nThe discussion with other reviewers and the authors' responses have provided a more comprehensive understanding of the paper's contributions and limitations.\n\nThe authors have committed to providing more detailed explanations in the final version of the paper, which will prevent potential misunderstandings.", "rebuttal": "Thanks for acknowledging our idea is clear and the experiments are comprehensive. We provide point-to-point responses to your concerns as follows:\n\n---\n\n**W1: The proof is only for SGD with momentum ($\\eta=1$ is also hard-coded).**\n\n**Reply:** \n- We would like to clarify that the learning rate $\\eta$ is just a scaling factor in our analysis, setting it to 1 only simplifies the notation and **does not lose the generality of our theorem at all**.\n- We focus on analyzing the effectiveness of decoupling itself in Sec. 3.3. **The effectiveness of preconditioning is discussed in Sec. 3.2 through numerical experiments.** Technically, it is quite challenging to rigorously bound the variance of multi-step Adam-optimized parameters. Nevertheless, our comprehensive experiments demonstrate the efficacy of the combination of preconditioning and decoupled momentum. Our ablation study demonstrate the effectiveness of using either preconditioning or decoupled momentum.\n\n---\n\n**W2: Theorem 3.5 does not mathematically connect this reduced variance to better performance on downstream metrics**\n\n**Reply:**  Thanks for pointing this out, we would like to provide further analysis as follows:\n\nWe let $M(\\theta)$ to represent the performance of model with parameter $\\theta$ where $M$ can be FA/RA/TA/MIA. Before we derive the variance bound of performance metric function $M(\\theta)$, we make the following assumption:\n\n**Assumption A.1.** The performance metric function $M$ is Lipschitz continuous: \n$ \\forall\\theta_1,\\theta_2, \\|M(\\theta_1)-M(\\theta_2)\\|\\leq L_M\\|\\theta_1-\\theta_2\\|$, \nwhere $L_M$ is a non-negative constant.\n\nFollowing a similar logic to 3.4, we have the following corollary:\n\n**Corollary A.2.** If Assumption A.1 holds, and the parameter variance satisfies \n\n$\\mathrm{Var}(\\theta)\\leq\\overline{\\mathrm{Var}}(\\theta)$,  then we have $\\mathrm{Var}(M(\\theta))\\leq L_M^2\\overline{\\mathrm{Var}}(\\theta)$.\n\n*Proof.*\nSimilar to the proof of Lemma 3.4, we have:\n$$\\mathrm{Var}(M(\\theta)) = \\frac{1}{2}\\iint p(\\theta_1)p(\\theta_2)\\left\\|M(\\theta_1) - M(\\theta_2)\\right\\|^2 d\\theta_1 d\\theta_2,$$\n\nwhere $\\theta_1, \\theta_2$ are independent copies of $\\theta$ with probability density $p(\\theta)$.\n\nBy $\\|M(\\theta_1)-M(\\theta_2)\\|\\leq L_M\\|\\theta_1-\\theta_2\\|$, we have:\n\n$$ \\mathrm{Var}(M(\\theta)) \\leq \\frac{L_M^2}{2}\\iint p(\\theta_1)p(\\theta_2)\\|\\theta_1 - \\theta_2\\|^2 d\\theta_1 d\\theta_2.$$\n\nSince $\\mathrm{Var}(\\theta)\\leq\\overline{\\mathrm{Var}}(\\theta)$, we have $\\frac{1}{2}\\iint p(\\theta_1)p(\\theta_2)\\|\\theta_1 - \\theta_2\\|^2 d\\theta_1 d\\theta_2 \\leq \\overline{\\mathrm{Var}}(\\theta)$. Therefore, we can conclude $\\mathrm{Var}(M(\\theta)) \\leq L_M^2 \\overline{\\mathrm{Var}}(\\theta)$.\n\n**Discussion.** The final unlearned parameters are usually the parameter updated by retaining loss at the last iteration $T$, i.e., $\\theta^D_{r,T}$ using decoupled momentum and $\\theta^S_{r,T}$ using shared momentum. \nAccording to Theorem 3.5, we have $\\overline{\\mathrm{Var}}(\\theta^D_{r,T}) \\leq \\overline{\\mathrm{Var}}(\\theta^S_{r,T})$. Let the upper bound of the performance variance $\\overline{\\mathrm{Var}}(M(\\theta))=L_M^2 \\overline{\\mathrm{Var}}(\\theta)$, we can derive $\\overline{\\mathrm{Var}}(M(\\theta^D_{r,T})) \\leq \\overline{\\mathrm{Var}}(M(\\theta^S_{r,T}))$. \nThus, **decoupled momentum can also induce less variability on performance metrics**.\n\n---\n\n**W3: The paper does not detail if the search budget was equal for the baselines versus DualOptim, which introduces two separate learning rates**\n\n**Reply:** Thanks for your constructive comment. Competitive baselines such as SaLUn and SFRon also use two separate learning rates but a shared momentum factor. In such cases, we fix the retaining learning rate $\\eta_r=1e-2$ when using SGD or $\\eta_r = 1e-4$ when using Adam, and search for the optimal forgetting learning rate in a wide range and a fine granularity. Specifically, **we start with a reasonable learning rate that induces effective unlearning, and carefully search for the optimal value in a similar order of magnitude**. For instance, if the starting value is 1e-4, the search range will be [5e-5, 6e-5, 7e-5, 8e-5, 9e-5, 1e-4, 1.1e-4, 1.2e-4, 1.3e-4, 1.4e-4, 1.5e-4]. Note that, since the reasonable starting learning rates of different methods vary a lot, we do not adopt the unified search range for all baselines. \n\nFor baselines with a unified learning rate, we follow a similar strategy as above to start with a reasonable learning rate and carefully search for the optimal one. In summary, **all baselines are carefully tuned to achieve the near-optimal performance**. \nWe will include this detail in the revised version.\n\n---\n\n**W4: The main results only compare DualOptim's alternating update schedule with baselines that sometimes use a joint optimization objective.**\n\n**Reply:** Thanks for pointing this out. In Sec. 4.3, we aim to compare our method with **unmodified baselines that were originally designed to use a joint optimization objective**. Nonetheless, we also include the results of their alternating variant for a more comprehensive comparison. The results of Table 14 of Appendix C.5 show that **DualOptim outperforms the original baselines and their alternating variants**. We will highlight this point in the main text of the revised manuscript to avoid confusion.\n\n---\n\n**Q1: Report the wall-clock time and peak memory usage for DualOptim compared to a key baseline on a representative task**\n\n**Reply:**  Thanks for your suggestion, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that **under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim).** Nonetheless, **under the LoRA configuration, the memory usage ratio is very close to 1:1** since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but **DualOptim takes 2% to 50% more time than Adam in general**.\n\nWhile the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA,  to significantly reduce the memory usage of optimizer states. As shown in Table 15, **unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters**. We believe the development of PEFT will also broaden the application scenarios of our method.\n\n**Remark:** For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, **the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.**\n\n***Time Usage (s)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |1366.47| 2034.98 | 1.49 | \n| Llama | ME+GD | 1163.95 | 1710.18 | 1.47 | \n| Llama w/ LoRA | IDK+AP | 1605.75 | 1939.18 | 1.21 | \n| Llama w/ LoRA | ME+GD | 1022.49 | 1045.55 | 1.02 | \n\n***Memory Usage (GB/GPU)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |59.84|86.52|1.45| \n| Llama | ME+GD | 59.83 | 86.40 | 1.44 |\n| Llama w/ LoRA | IDK+AP |31.44|30.62|0.97| \n| Llama w/ LoRA | ME+GD | 30.63 | 30.62 | 1.00 | \n\n---  \n\n**Q2: Detail the hyperparameter tuning budget for both DualOptim and the baselines to ensure a fair comparison** \n\n**Reply:**  Please see the response to W3.\n\n---  \n\n**Q3: address the mismatch between your theory (proven for SGD) and your algorithm (which uses Adam)** \n\n**Reply:**  Please see the response to W1.\n\n---\n\n**We hope our responses can address your concerns in the review above and look forward to your feedback.**"}, {"review_id": "XluPDIsplF", "reviewer": "Reviewer_UZKz", "summary": "This paper introduces DualOptim, a novel method designed to enhance the efficacy and stability of approximate machine unlearning (MU) methods. DualOptim addresses the instability and hyperparameter sensitivity of existing MU techniques by employing two key innovations: 1. adaptive learning rates for the forgetting objective and 2. decoupled momentum updates for the forgetting and retaining objectives. The method is validated through extensive experiments across diverse tasks, including image classification, image generation, and LLMs. Results demonstrate that DualOptim significantly improves the performance and stability of existing MU algorithms, making it a versatile and impactful solution.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper is technically sound, with strong theoretical foundations supporting the proposed method. The decoupling of momentum is rigorously analyzed, and its impact on variance reduction is well-justified.\n2. Extensive experiments across multiple domains (image classification, image generation, and LLMs) demonstrate the generalizability of DualOptim.\n3. The paper is clearly written and structured, with detailed explanations of the methodology and experimental setup.\n\nWeaknesses\n1. The use of decoupled momentum introduces additional memory and computational overhead, particularly for large-scale models like LLMs. While the authors discuss this limitation, further optimization strategies could be explored.\n2. While the paper demonstrates the effectiveness of DualOptim, it lacks a detailed analysis of the sensitivity of its own hyperparameters (e.g., learning rates for forgetting and retaining objectives). Additionally, the experiments compare DualOptim with other methods using their default hyperparameters, which may not represent the optimal performance of these baselines. A fairer comparison would involve tuning all methods under the same conditions.", "was_revised": true, "final_justification": "The additional experiments were helpful and have addressed my concerns. They enhance the completeness of the submission, but I consider them expected rather than transformative additions. I therefore keep the overall rating unchanged, while increasing the confidence score.", "rebuttal": "Thanks for acknowledging that our paper is technically sound, well-written and the experiments are extensive. We provide point-to-point responses to your concerns as follows:\n\n---\n\n**W1: While the authors discuss this limitation, further optimization strategies could be explored**\n\n**Reply:** For your reference, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that **under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim).** Nonetheless, **under the LoRA configuration, the memory usage ratio is very close to 1:1** since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but **DualOptim takes 1~1.5x more time than Adam in general**.\n\nWhile the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA,  to significantly reduce the memory usage of optimizer states. As shown in Table 15, **unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters**. We believe the development of PEFT will also broaden the application scenarios of our method.\n\n**Remark:** For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, **the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.**\n\n***Time Usage (s)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |1366.47| 2034.98 | 1.49 | \n| Llama | ME+GD | 1163.95 | 1710.18 | 1.47 | \n| Llama w/ LoRA | IDK+AP | 1605.75 | 1939.18 | 1.21 | \n| Llama w/ LoRA | ME+GD | 1022.49 | 1045.55 | 1.02 | \n\n***Memory Usage (GB/GPU)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |59.84|86.52|1.45| \n| Llama | ME+GD | 59.83 | 86.40 | 1.44 |\n| Llama w/ LoRA | IDK+AP |31.44|30.62|0.97| \n| Llama w/ LoRA | ME+GD | 30.63 | 30.62 | 1.00 | \n\n---\n\n**W2: It lacks a detailed analysis of the sensitivity of its own hyperparameters. Additionally, the experiments compare DualOptim with other methods using their default hyperparameters, which may not represent the optimal performance of these baselines.**\n\n**Reply:**\n- Thanks for your suggestion. We report the performance of SFRon and SFRon+DualOptim with different forget learning rates below (we fix retaining learning rate to 0.01). It can be observed that **our method significantly stabalizes the unlearning process and achieves smaller gap to retrained models in a wide range of hyperparameter choices.**\n- We would like to clarify that for all baselines, we search for their optimal learning rates in a wide range and a fine granularity (all implementation details are shown in Appendix B). For SalUn and SFRon which use separate learning rates for retain and forget sets, we fix the retaining learning rate to a reasonable value and tune the forgetting learning rate. Specifically, **we start with a reasonable learning rate that induces effective unlearning, and search the optimal one at a granularity that is an order of magnitude smaller than the starting value**. For instance, if the starting value is 1e-4, the search range will be [5e-5, 6e-5, 7e-5, 8e-5, 9e-5, 1e-4, 1.1e-4, 1.2e-4, 1.3e-4, 1.4e-4, 1.5e-4]. Note that, since the reasonable starting learning rates of different methods vary a lot, we do not adopt the unified search range for all baselines. Therefore, **all baselines are carefully tuned to achieve the near-optimal performance**. We will include this detail in the revised version.\n\n***SFRon***\n| Forget lr | 0.027 | 0.028 | 0.029 | 0.03 | 0.031 | 0.032 | 0.033 | 0.034 | 0.035 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Avg Gap | 2.56 | 2.03 | 1.49 | 0.98 | 0.51 | 0.54 | 0.52 | 0.9 | 1.19 | \n| Avg Std | 0.99 | 1.57 | 2.03 | 2.43 | 2.33 | 2.37 | 2.23 | 2.20 | 2.16 | \n\n***SFRon+DualOptim***\n| Forget lr | 6e-5 | 7e-5 | 8e-5 | 9e-5 | 1e-4 | 1.1e-4 | 1.2e-4 | 1.3e-4 | 1.4e-4|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Avg Gap | 1.83 | 1.43 | 1.04 | 0.72 | 0.44 | 0.52 | 0.52 | 0.76 | 1.19 | \n| Avg Std | 0.49 | 0.59 | 0.61 | 0.67 | 0.66 | 0.72 | 0.68 | 0.73 | 0.77 | \n\n---\n\n**We hope our responses can address your concerns in the review above and look forward to your feedback.**"}, {"review_id": "xfbmd7wzV6", "reviewer": "Reviewer_XHDw", "summary": "This paper tackles the instability that plagues many machine-unlearning algorithms by introducing DualOptim, a two-optimizer scheme: one adaptive optimizer for the forget objective and another for the retain objective, with their momentum terms fully decoupled. The authors supply a variance-bound analysis explaining why decoupled momentum should reduce parameter drift, and they validate the method on image-classification, image-generation, and large-language-model tasks. Across these settings, DualOptim consistently narrows the gap to full retraining and lowers variance, suggesting it is a practical “plug-and-play” upgrade for existing MU pipelines.", "questions": "1. Are there examples where decoupling the optimizers actually hurts convergence?\n\n2. Can you provide concrete measurements of memory usage and run-time overhead when scaling to large LLMs? How do these costs compare with single-optimizer baselines?\n\n3. Have you evaluated DualOptim against automatic hyper-parameter scheduling techniques designed to stabilize MU?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. DualOptim effectively addresses the hyperparameter sensitivity issue prevalent in existing MU methods, providing significantly improved stability across diverse deployment scenarios.\n\n2. The approach demonstrates consistent performance gains across multiple domains including image classification, generation, and LLMs, showcasing its broad applicability.\n\n3. The solution combines both theoretical foundations (adaptive learning rates and decoupled momentum) and extensive empirical validation, ensuring reliable unlearning performance.\n\n**Weaknesses**\n\n1. Using different optimizers or decoupled momentum for multi-objective training is familiar from multitask and bilevel learning. The paper does not clearly separate its contribution from those earlier ideas.\n\n2. Theoretical assumptions like uncorrelated forget/retain gradients may not hold in practice, and the paper provides little guidance on when DualOptim could underperform.\n\n3. Maintaining two optimizers doubles state storage and may slow training, yet the main text offers no quantitative overhead analysis, especially for large models.", "was_revised": true, "final_justification": "This paper presents a technically sound and well-validated approach to improving the stability of machine unlearning via a dual-optimizer framework. While some aspects build on familiar ideas from multitask and bilevel optimization, the method’s targeted design for conflicting objectives in MU and its consistent performance gains justify acceptance.", "rebuttal": "We really appreciate you acknowledge that our work is sufficient for acceptance. We provide point-to-point responses to your concerns as follows:\n\n---\n\n**W1: The paper does not clearly separate its contribution from multi-task and bilevel learning**\n\n**Reply:** Thanks for pointing this out. We will add a brief discussion in our revised manuscript. From the optimisation perspective, empirical MU methods aims to optimise two objectives (1) minimize the loss on the retain set; (2) maximize the loss on the forget set. The second objective is unbounded, which is different from most multi-objective training problems which aim to optimize a single objective function that is a combination (e.g., a weighted sum) of the losses from all tasks. Therefore, MU methods are quite different from multi-objective training ones. Bilevel learning is defined by a hierarchical structure with an upper-level and a lower-level optimization problem. The most popular and competitive MU methods do not formulate the MU tasks as bilevel optimization problems.\n\nAlthough these three tasks share some commonalities, **our method is specifically designed to mitigate the unstability caused by two objectives with conflict in MU tasks.**\n\n---\n\n**W2: Theoretical assumptions like uncorrelated forget/retain gradients may not hold in practice, and the paper provides little guidance on when DualOptim could underperform.**\n\n**Reply:** We are sorry about the typo in line 153, **we will replace $\\rho(\\widehat{g_{f,t_1}}, \\widehat{g_{r,t_2}})\\simeq 0$ with $\\rho(\\widehat{g_{f,t_1}}, \\widehat{g_{r,t_2}}) \\leq o(\\tau) \\simeq 0$ in the revision**. We only assume that the correlation between forget gradient and retain gradient in different time steps is negligible compared with the correlation among forget gradients or the correlation among the retain gradients.\nIn the proof, we only utilize the assumption $\\rho\\(\\widehat{g_{f,t_1}}, \\widehat{g_{r,t_2}}\\) \\leq 0$ for notation simplicity. This encompasses the situation that the forgetting gradient is negatively correlated with the retaining gradient. Therefore, **it does not change the conclusion of our theorem.** \n\nMoreover, to make the analysis more rigorous, we can also replace the last assumption as $\\rho(\\widehat{g_{f,t_1}}, \\widehat{g_{r,t_2}}) \\leq \\xi \\ll \\tau$, \nthe conclusion still hold, because the term of $\\overline{\\mathrm{Var}}$ will be dominated by the term containing $\\tau$. \nThanks for pointing it out again, we will revise this part accordingly.\n\nAs for the second point, according to Eq. 5 and Theorem 3.5, when the momentum coefficient $\\alpha=0$, i.e., no momentum, DualOptim is equivalent to a single optimizer. However, modern optimizers always incorporate momentum for better optimization. Additionally, **our experiments show that our method consistently provides better performance in diverse tasks**.\n\n---\n\n**W3: The main text offers no quantitative overhead analysis, especially for large models.**\n\n**Reply:**  Thanks for your suggestion, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that **under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim).** Nonetheless, **under the LoRA configuration, the memory usage ratio is very close to 1:1** since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but **DualOptim takes 2% to 50% more time than Adam in general**.\n\nWhile the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA,  to significantly reduce the memory usage of optimizer states. As shown in Table 15, **unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters**. We believe the development of PEFT will also broaden the application scenarios of our method.\n\n**Remark:** For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, **the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.**\n\n***Time Usage (s)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |1366.47| 2034.98 | 1.49 | \n| Llama | ME+GD | 1163.95 | 1710.18 | 1.47 | \n| Llama w/ LoRA | IDK+AP | 1605.75 | 1939.18 | 1.21 | \n| Llama w/ LoRA | ME+GD | 1022.49 | 1045.55 | 1.02 | \n\n***Memory Usage (GB/GPU)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |59.84|86.52|1.45| \n| Llama | ME+GD | 59.83 | 86.40 | 1.44 |\n| Llama w/ LoRA | IDK+AP |31.44|30.62|0.97| \n| Llama w/ LoRA | ME+GD | 30.63 | 30.62 | 1.00 | \n\n---\n\n**Q1: Are there examples where decoupling the optimizers actually hurts convergence?**\n\n**Reply:** We did not observe such examples so far.\n\n---\n\n**Q2: Can you provide concrete measurements of memory usage and run-time overhead when scaling to large LLMs? How do these costs compare with single-optimizer baselines?**\n\n**Reply:** Please see the response to W3.\n\n---\n\n**Q3:Have you evaluated DualOptim against automatic hyper-parameter scheduling techniques designed to stabilize MU?**\n\n**Reply:** It should be noted that the hyperparameters of all evaluated methods are tuned through a **sophisticated search** (details in the responses to W2 for reviewer UZKz and W3 for reviewer wno1), so the results exhibit the optimal performance of the evaluated methods. Moreover, automatic scheduling techniques are embedded in specific MU methods [1,2] and are orthogonal to our method. Therefore, they are not included in our comparison.\n\n---\n\n**We hope our responses can address your concerns.**\n\n[1] Chongyu Fan et al. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification\nand generation. ICLR 2024.\n\n[2] Zhehao Huang et al. Unified gradient-based machine unlearning with remain geometry enhancement. NeurIPS 2024."}, {"review_id": "IzcpVQcYKo", "reviewer": "Reviewer_YcdJ", "summary": "This paper proposes a “dual-optimizer” strategy for approximate machine unlearning (MU). During unlearning, gradients from the forget loss are updated with Adam, while gradients from the retain loss continue with the model’s original optimizer (typically SGD); their momentum buffers are fully decoupled. The authors prove a tighter bound on parameter-variance under decoupled momentum and demonstrate consistent gains and markedly lower run-to-run variance across MU on image classifiers, class-wise unlearning in generative diffusion model, and 7 B-parameter LLM.", "questions": "- Could the authors quantify wall-clock overhead and GPU memory increase on the LLaMA-2 experiment?\n- How sensitive is DualOptim when the pre-training optimizer is already adaptive (e.g., AdamW, Adan)?\n- Typo in Section A.2 Proof of Theorem 3.5 Equation (11) Decoupled Momentum, $\\mathbf{m}^S_{f,i}$ should be $\\mathbf{m}^D_{f,i}$", "limitations": "Mostly yes. However, it may be beneficial to include more discussion on the conditions under which DualOptim exhibits limited performance and the reasons behind it.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The idea is simple yet effective, it requires minimal code change to deploy in the real world.\n- The authors provides very comprehensive empirical validation with convincing details on implementation and experimental setup. Open sourcing the code and results will help the unlearning community greatly. \n- The theoretical analysis provides a clear variance bound links the decoupled-momentum idea to stability, providing more than just empirical intuition.\n\nWeaknesses\n- Assumption 3.2 treats forget-retain gradient correlation as negligible, yet Figure 4’s cosine-similarity plot is not a rigorous test of that claim. Firstly, from intuition, forgetting objective and retaining objective is possibly highly negatively correlated during a whole training session (across iterative updates). Secondly, cosine similarity treats the *coordinates* of a gradient vector as “samples” and asks whether the two vectors are roughly orthogonal **within a single timestep**. The correlation in Assumption 3.2, by contrast, is about how two *random variables* (the gradient estimates produced by data sampling) co-vary **across different draws**. These are different statistical objects. Also, due to curse of dimensionality, the reported value in Figure 4 may simply reflect high dimensionality, not genuine near-zero correlation. \n- The paper’s overall idea is slightly too incremental, the impact at conceptual level is limited. (but this weakness is supplemented by the comprehensiveness of the empirical result, which I as a reviewer do appreciate.)\n- More advanced optimizers and their combinations could be included in the experimental/ablation study setup.", "was_revised": true, "final_justification": "I have read authors’ responses and discussions from all other reviewers too, I believe the authors’ responses have addressed most of my concerns. Despite the simplicity of the idea, I am satified with the comprehensive, convincing experimental results. Thus, I am happy to keep my positive rating.", "rebuttal": "Thanks for acknowledging that our method is simple yet effective, and the experiments are comprehensive. We provide the point-to-point responses to your concerns as follows:\n\n---\n\n**W1: Assumption 3.2 treats forget-retain gradient correlation as negligible, yet Figure 4’s cosine-similarity plot is not a rigorous test of that claim.**\n\n**Reply:** We are sorry about the typo in line 153, we will replace $\\rho(\\widehat{g_{f,t_1}},\\widehat{g_{r,t_2}})\\simeq 0$ with $\\rho(\\widehat{g_{f,t_1}},\\widehat{g_{r,t_2}}) \\leq o(\\tau) \\simeq 0$ in the revision. We only assume that the correlation between forget gradient and retain gradient in different time steps is negligible compared with the correlation among forget gradients or the correlation among the retain gradients. In the proof, we only utilize the assumption $\\rho(\\widehat{g_{f,t_1}}, \\widehat{g_{r,t_2}}) \\leq 0$ for notation simplicity. This encompasses the situation that the forgetting gradient is negatively correalted with the retaining gradient. Therefore, **it does not change the conclusion of our theorem.**\n\nMoreover, to make the analysis more rigorous, we can also replace the last assumption as $\\rho(\\widehat{g_{f,t_1}},\\widehat{g_{r,t_2}}) \\leq \\xi \\ll \\tau$, the conclusion still hold, because the term of $\\overline{\\mathrm{Var}}$ will be dominated by the term containing $\\tau$.\n\nThanks for pointing it out again, we will revise this part accordingly.\n\n---\n\n**W2: The paper’s overall idea is slightly too incremental, the impact at conceptual level is limited.**\n\n**Reply:** We would like to highlight that **our method is simple but effective.** We believe that the coupled momentum factor is one of the fundamental reasons leading to the instability of MU methods. Empirical and theoretical evidences demonstrate effectiveness of using dual optimizers in stabilizing the MU process and improving the unlearning performance. Furthermore, our method is not task-specific, making it a versatile approach to empower existing MU algorithms. Therefore, we believe that our method can be broadly applied.\n\n---\n\n**W3: More advanced optimizers and their combinations could be included.**\n\n**Reply:** In Table 5, we include the results of different combinations of Lion optimizer and SGD. In addition, we report the results of the combinations of recently popular **Muon** [1] and SGD under the same setting of Table 5 below. We can observe that **Muon also exhibits low variability, but does not achieve the optimal performance** compared with Adam, while the best combination: **Adam(F)+SGD(R)** reduces the average gap to **0.44** with an average std of **0.66**.\n\n| Optimizer | FA | RA | TA | MIA | Avg Gap | Avg Std | \n|---|---|---|---|---|---|---|\n| SGD(F) + Muon(R) | 95.74±0.40 | 99.61±0.03 | 94.04±0.09 | 84.03±0.77 | 2.37 | 0.32 |\n| Muon(F) + Muon(R) | 95.12±0.54 | 99.64±0.01 | 93.84±0.10 | 82.77±0.60 | 1.94 | 0.31 | \n| Muon(F) + SGD(R) | 94.57±1.12 | 99.93±0.01 | 94.13±0.12 | 72.46±1.01 | 1.01 | 0.57 | \n\n---\n\n**Q1: Could the authors quantify wall-clock overhead and GPU memory?**\n\n**Reply:**  Thanks for your suggestion, we report the wall-clock time and peak memory usage of Adam and DualOptim on Llama 2 with full parameter tuning and LoRA below. We can observe that **under the configuration of full parameter tuning, the memory usage ratio is closer to 1 (Adam) : 1.5 (DualOptim).** Nonetheless, **under the LoRA configuration, the memory usage ratio is very close to 1:1** since the number of trainable parameters is significantly reduced. Additionally, the time usage ratio varies across different configurations and specific MU algorithms, but **DualOptim takes 1~1.5x more time than Adam in general**.\n\nWhile the memory overhead introduced by our method is not negligible for LLMs, we can adopt parameter-efficient fine-tuning (PEFT) methods, such as LoRA,  to significantly reduce the memory usage of optimizer states. As shown in Table 15, **unlearning with LoRA can also induce competitive performance compared to unlearning with full parameters**. We believe the development of PEFT will also broaden the application scenarios of our method.\n\n**Remark:** For LoRA, we used 1× H20 GPU; for full parameter tuning, we used 2× H20 GPUs. We implement DualOptim based on bitsandbytes (bnb) library, because it can be easier to maintain the optimizer states in the framework of transformers library, while Adam is implemented based on original pytorch. Since bnb has specially optimized the implementation of optimizers and LoRA significantly decreases the number of trainable parameters, **the memory usages of DualOptim is even smaller than that of Adam under the LoRA configuration.**\n\n***Time Usage (s)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |1366.47| 2034.98 | 1.49 | \n| Llama | ME+GD | 1163.95 | 1710.18 | 1.47 | \n| Llama w/ LoRA | IDK+AP | 1605.75 | 1939.18 | 1.21 | \n| Llama w/ LoRA | ME+GD | 1022.49 | 1045.55 | 1.02 | \n\n***Memory Usage (GB/GPU)***\n| Model | MU method | Adam | DualOptim | Ratio |\n|---|---|---|---|---|\n| Llama | IDK+AP |59.84|86.52|1.45| \n| Llama | ME+GD | 59.83 | 86.40 | 1.44 |\n| Llama w/ LoRA | IDK+AP |31.44|30.62|0.97| \n| Llama w/ LoRA | ME+GD | 30.63 | 30.62 | 1.00 | \n\n---\n\n**Q2 : How sensitive is DualOptim when the pre-training optimizer is already adaptive?**\n\n**Reply:** We included the results of the model pretriained with Adam in Table 17. The results show that when the pre-training optimizer is Adam, the best combination is Adam(F) + Adam(R), which is consistent with the clam that we use **the same optimizer as in pretraining to maintain the performance on the retain set** and use **the optimizer with adaptive learning rates to handle the large gradient variation for the forget loss**.\n\n---\n\n**Q3 : Typo in Section A.2 Proof of Theorem 3.5 Equation (11)**\n\n**Reply:** Thanks for pointing this out. We will correct it in the revision.\n\n---\n\n**We hope our responses can address your concerns in the review above and look forward to your feedback.**\n\n[1] Keller Jordan et al. Muon: An optimizer for hidden layers in neural networks."}]

Author rebuttal:

Meta-review: This paper proposes a new method for resolving the instability of machine unlearning, by decoupling the "forget" and "retain" optimizations. Overall, reviewers found the paper easy to read and pretty convincing experimentally. The authors should make sure to include the results and edits promised in the rebuttal into the final paper.
Decision: Accept (poster)