Title: DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
Venue: NeurIPS 2025
Accepted: yes
Keywords: machine unlearning
Abstract: Existing machine unlearning (MU) approaches exhibit significant sensitivity to hyperparameters, requiring meticulous tuning that limits practical deployment. In this work, we first empirically demonstrate the instability and suboptimal performance of existing popular MU methods when deployed in different scenarios. To address this issue, we propose Dual Optimizer (DualOptim), which incorporates adaptive learning rate and decoupled momentum factors. Empirical and theoretical evidence demonstrates that DualOptim contributes to effective and stable unlearning. Through extensive experiments, we show that DualOptim can significantly boost MU efficacy and stability across diverse tasks, including image classification, image generation, and large language models, making it a versatile approach to empower existing MU algorithms.

DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers

Abstract

Existing machine unlearning (MU) approaches exhibit significant sensitivity to hyperparameters, requiring meticulous tuning that limits practical deployment. In this work, we first empirically demonstrate the instability and suboptimal performance of existing popular MU methods when deployed in different scenarios. To address this issue, we propose Dual Optimizer ( DualOptim ), which incorporates adaptive learning rate and decoupled momentum factors. Empirical and theoretical evidence demonstrates that DualOptim contributes to effective and stable unlearning. Through extensive experiments, we show that DualOptim can significantly boost MU efficacy and stability across diverse tasks, including image classification, image generation, and large language models, making it a versatile approach to empower existing MU algorithms. Codes are available at https://github.com/CityUMLO/DualOptim.

1 Introduction

Recent advancements in machine unlearning (MU) research have established it as a crucial technique for utilizing pretrained models while addressing various trustworthy challenges in various applications [1, 2]. These MU methods have two major categories: exact MU [3] and approximate MU [4]. Exact MU requires retraining a model on a dataset excluding forgetting samples from scratch, which is computationally prohibitive for large-scale models and datasets. Therefore, we study approximate MUmethods in this work, with focuses on their efficacy and stability.

Most existing approximate MU methods [5, 6, 7, 8, 9] aim to maximize loss on forget samples while preserving performance on retain samples. While effective, these methods exhibit significant sensitivity to hyperparameter choices, with optimal configurations requiring meticulous tuning that varies substantially across datasets and forgetting scenarios. This dependency complicates their practical deployment. As illustrated in Figure 1, we empirically demonstrate that state-of-the-art MU approaches [6, 7, 8] exhibit suboptimal performance or instability. These limitations highlight the critical need for techniques to improve the effectiveness and stability of MU methods.

In this work, we first reveal the unstable performance of existing MU methods when they are applied in different scenarios. Considering the need to optimize two objectives on the forgetting

0

200

400

600

800

1000

1200

1400

Iteration

86

88

90

92

94

96

98

100

SFRon

SalUn

SCRUB

Retrain

0

200

400

600

800

1000

1200

1400

Iteration

96

97

98

99

100

SFRon

SalUn

SCRUB

Retrain

0

200

400

600

800

1000

1200

1400

Iteration

89

90

91

92

93

94

95

SFRon

SalUn

SCRUB

Retrain

0

200

400

600

800

1000

1200

1400

Iteration

60

65

70

75

80

85

90

95

SFRon

SalUn

SCRUB

Retrain

(d) MIA

Figure 1: Unlearning process of MU baselines. SFRon [7], SalUn [6] and SCRUB [8] are adopted as the baselines. The metrics are those mentioned in Sec. 3.1. All results are obtained from unlearning 10% random subset of CIFAR-10 on ResNet-18. The solid lines and shadows denote the mean and standard deviation across 5 trials with different random data. The hyperparameters of different methods are selected based on minimizing the averaging gap between retraining and them across 5 trials. The red dashed lines denote the final performance of retraining as a reference.

and the retaining samples, we then propose Dual Optimizer ( DualOptim ) to address this challenge. Specifically, we employ an optimizer with adaptive learning rate, such as Adam [10, 11], to optimize the forgetting objective, while utilizing a separate optimizer for the retaining objective. Compared with existing methods, DualOptim decouples the momentum terms for the two objectives by using and updating them separately. In addition, DualOptim is a plug-and-play solution that can be easily integrated into existing MU algorithms.

Our theoretical analysis validates that decoupling momentum terms in two separate optimizers can help decrease the variance of model parameters, indicating a more stable performance. Through comprehensive experiments, we validate the effectiveness of DualOptim in improving the performance of existing MU methods across various tasks, including image classification, image generation, and large language models. Our proposed technique is generic and capable of pushing the state-of-the-art performance on multiple tasks. We summarize the contributions of this paper as follows:

We introduce Dual Optimizer ( DualOptim ) to enhance the efficacy and stability of MU methods by incorporating an adaptive learning rate and decoupled momentum. DualOptim can be seamlessly integrated into existing MU algorithms.

We provide empirical and theoretical analyses to demonstrate the contribution of DualOptim in improving unlearning performance and stability.

Comprehensive experiments across diverse scenarios such as image classification, image generation, and large language models validate the effectiveness of DualOptim in boosting and stabilizing the performance of MU methods.

2 Related Works

Machine Unlearning (MU). MUtargets the need to remove specific data influences from pretrained models [12, 13, 14], while complying with privacy requirements tied to differential privacy [3, 15, 16, 17, 18]. Initially developed on linear classifiers with convex loss objective functions, exact MU[3, 15, 19, 20] allowed precise data removal under privacy budgets to counter privacy attacks. However, exact MU cannot be applied to deep neural networks due to their non-convex loss functions and prohibitive computational cost for full retraining. In this context, approximate MU [4, 21] was proposed to address this issue by fine-tuning the model to achieve the forgetting effect. Despite improved efficiency, approximate MU may cause catastrophic performance declines on retaining data [21, 22]. Recent methods incorporate fine-tuning [6, 8, 23], sparsity regularization [24], knowledge distillation [8, 25], saliency map [6, 7] and alternative updating [7] to better balance efficient forgetting and model utility. However, as illustrated in Figure 1, these methods generally exhibit suboptimal performance or high hyperparameter sensitivity, underscoring the necessity to develop a method to enhance efficacy and stability during unlearning.

MUfor Multiple Tasks. Besides classification, MU has broad applications for multiple tasks. For example, recent text-conditional image generation models have showcased their ability to produce images that closely align with textual descriptions [26, 27, 28, 29]. However, significant security and privacy concerns have been raised [1, 30, 31], necessitating the application of MU methods to these

models. While early works [30, 32, 33, 34] focuses on concept deletion in diffusion models, recent studies [6, 7, 35] improve their performance and propose methods applicable to more general image generators. Another notable example is large language models (LLMs) , which have demonstrated remarkable capabilities [36, 37] but also face privacy and copyright issues like retaining unauthorized content [38, 39, 40, 41]. To address these concerns, MU methods have been employed to fine-tune LLMs [2, 9, 42, 43] to effectively and efficiently achieve data forgetting. However, all these methods struggle to achieve a balance between model utility and forget effectiveness. In addition, intensive hyperparameter tuning is expected for optimal performance, bringing challenges for practitioners.

In summary, MU has broad applications across various tasks. However, there are some common issues with existing methods, including high hyper-parameter sensitivity and the utility-forgetting trade-offs. In this work, we propose DualOptim as a generic solution to enhance the efficacy and stability of MU across different tasks.

3 Methodology

3.1 Preliminary

Let D = { z i } N i =1 denote the training set for pretraining. The subset of the training set we aim to forget during unlearning is known as the forget set D f ⊂ D , and its complement, D r = D\D f is the retain set . In the context of MU, we denote the parameters of pretrained model as θ o ∈ R d , which is trained on D . Consistent with previous studies [6, 7, 21, 24], Retraining is considered the gold standard for MU, where the model parameters θ r ∈ R d are trained from scratch on D r . However, retraining is computationally intensive or even infeasible, especially for large models and datasets. This poses a significant challenge in the practical applications of MU.

We focus on approximate MU in this work, its primary objective is to obtain an unlearned model, referred to as θ u ∈ R d , from θ o on D f and D r so that it can serve as an accurate and computationally efficient alternative to the retrained model θ r . Mathematically, MU aims to solve the optimization problem: min θ L f ( D f , θ ) + L r ( D r , θ ) , where L f and L r denote forget and retain loss objective functions, respectively. In practice, L r is usually the same as the loss objective function in pretraining and L f is the opposite, so MU aims to improve the performance on D r while degrading the performance on D f . To avoid confusion, we call L f ( D f , θ ) , L r ( D r , θ ) that we aim to minimize forget loss and retain loss , respectively. In addition, many MU algorithms [6, 7, 8] update θ by alternately optimizing L f ( D f , θ ) and L r ( D r , θ ) for better performance. The generic pipeline of MU is presented in Algorithm 1. For notation simplicity, we use g f := ∇ θ L f ( D f , θ ) and g r := ∇ θ L r ( D r , θ ) to denote the gradients of the forget loss and the retain loss, respectively. In mini-batch updates, we use ̂ g f and ̂ g r to denote the corresponding stochastic gradients.

The numerical analysis in this section is based on classification tasks. We adopt the same evaluation schemes as [7]. That is, we use the accuracy on the forget set (FA), the retain set (RA), test set (TA) and the success rate of membership inference attack (MIA) [44] 2 on the forgetting set to indicate model utility and the forgetting effectiveness. A competitive unlearning method should have stable performance and small gaps with retraining in all these four evaluation schemes.

In the following subsections, we first illustrate the challenges associated with MU and propose corresponding approaches to address them. Ultimately, we introduce DualOptim , which integrates these proposed techniques to enhance both efficacy and stability in MU.

3.2 Adaptive Learning Rate Enables Stable Forgetting

Despite the effectiveness of existing MU methods, achieving satisfactory unlearning performance often relies on carefully selecting hyperparameters. The optimal hyperparameters can vary significantly across different forget sets, complicating the practical applications of MU algorithms. As shown in Figure 2 (a)-(d), we use the same hyperparameters for SFRon [7], the best-performed baseline in our evaluation, with the default SGD optimizer for 5 trials of different randomly sampled forget sets. We defer more results of other existing methods to Appendix C.2. Despite being intensively tuned, the unified hyperparameters exhibit quite unstable performance across different forget sets during unlearning. This underscores that a hyperparameter configuration effective for certain forget sets may

FA

0

200

400

600

800

1000

1200

1400

Iteration

86

88

90

92

94

96

98

100

SGD

Adam

DualOptim

Retrain

(a) Forget Accuracy (FA)

0

200

400

600

800

1000

1200

1400

Iteration

96

97

98

99

100

RA

SGD

Adam

DualOptim

Retrain

0

200

400

600

800

1000

1200

1400

Iteration

89

90

91

92

93

94

95

TA

SGD

Adam

DualOptim

Retrain

(c) Test Accuracy (TA)

0

200

400

600

800

1000

1200

1400

Iteration

60

65

70

75

80

85

90

95

MIA

SGD

Adam

DualOptim

Retrain

(d) MIA

0

200

400

600

800

1000

1200

1400

Iteration

0

20

40

60

80

100

Gradient Norm

SGD Forget

SGD Retain

(a) SGD

0

200

400

600

800

1000

1200

1400

Iteration

0

20

40

60

80

100

Gradient Norm

(b) Adam

0

200

400

600

800

1000

1200

1400

Iteration

0

20

40

60

80

100

Gradient Norm

DualOptim Forget

DualOptim Retain

(c)

DualOptim (Ours)

not be suitable for others. When dealing with a new forget set, even if it is sampled in the same way, the need for precise hyperparameter tuning can pose challenges for practitioners.

Figure 3 (a) demonstrates the magnitude of the gradients on the forget set and the retain set when using SGD optimizer. We can clearly see that the gradients on the forget set have significantly larger magnitudes with substantial variations compared with the ones on the retain set. In this context, it is crucial to adaptively adjust the learning rate to handle different gradient magnitudes. Therefore, we use optimizers with preconditioners, such as Adam [10, 11] to evade tricky hyperparameter tuning for the learning rate. Specifically, Adam employs adaptive learning rates based on the historic gradient magnitudes, making it suitable to handle large gradient magnitude variations during unlearning.

The observations in Figure 2 (a)-(d) and Figure 3 (b) suggest that Adam provides more stable performance and induces smaller gradient norm across different forget sets. However, despite enhanced stability, the noticeable performance gap with retraining, especially in the metric of membership inference attack (MIA), indicates that Adam only achieves suboptimal unlearning efficacy. We need a mechanism to improve performance while ensuring the stability of the method.

3.3 Decoupled Momentum for Enhanced Stability in Machine Unlearning

Inspired by the disparities in the magnitudes of ̂ g f and ̂ g r , along with their non-positive correlation observed in Figure 3 and 4, we introduce decoupled momentum , which employs two separate momentum terms dedicated to the forget loss L f and the retain loss L r , respectively. This approach ensures that the momentum update for the forget loss remains unaffected by the gradients of the retaining loss, and vice versa. By decoupling the momentum terms in this manner, we can more effectively optimize the distinct processes of forgetting and retaining, leading to enhanced stability and performance. As illustrated in Figure 2 (a)-(d) (represented by the green lines) and Figure 3 (c), the combination of adaptive learning rate with decoupled momentum not only stabilizes the unlearning process but also leads to improved unlearning efficacy. Note that as presented in Table 5 and Table 6 of Appendix C.2, although other methods like SalUn [6] and SCRUB [8] perform more stable than SFRon, they only achieve suboptimal unlearning performance. Nevertheless, our method can also improve their performance.

Besides empirical findings, we provide a theoretical analysis to elucidate how decoupled momentum contributes to stability. Before the detailed analyses, we first establish the following assumptions.

Assumption 3.1. (Stochastic Gradient Condition) For all time steps t = 0 , . . . , T -1 , the stochastic gradients of the forget loss ̂ g f,t and retain loss ̂ g r,t satisfy:

where g f,t := ∇ θ t L f ( D f , θ t ) and g r,t := ∇ θ t L r ( D r , θ t ) are the full-batch gradients with model parameter θ t at the time stamp t . glyph[epsilon1] f,t and glyph[epsilon1] r,t are batch noises with zero mean and a bounded variance: there exists a minimal σ 2 ≥ 0 such that Var( glyph[epsilon1] f,t ) ≤ σ 2 , Var( glyph[epsilon1] r,t ) ≤ σ 2 for all t .

Assumption 3.2. (Correlation Bounds) The correlation between the stochastic gradients from the same function in different time steps is bounded while the correlation between stochastic gradients from different functions is non-positive. That is to say, ∃ τ ∈ [0 , 1] such that:

glyph[negationslash]

The assumption ∀ t 1 , t 2 , ρ ( ̂ g f,t 1 , ̂ g r,t 2 ) ≤ o ( τ ) glyph[similarequal] 0 is motivated by the observations in Figure 4 of Appendix C.1 and for sake of notation simplicity. Our analyses can be easily extended to the cases if we use a small constant to bound this correlation.

Assumption 3.3. (Lipschitz Smoothness) The loss functions L f and L r are both L -smooth:

We assume the same Lipschitz constant for L f and L r because (1) D f and D r are from similar distributions; (2) L f and L r are usually opposite functions.

We now consider the SGD update scheme with a shared or decoupled momentum factor. Since we alternately update the parameters based on the stochastic gradients from L f and L r , we use { ( m S f,t , m S r,t ) } T -1 t =0 to denote the momentum factors after using the gradients from the forget loss and the retain loss, respectively, for time stamp t when we are using the shared momentum. Similarly, we use { ( m D f,t , m D r,t ) } T -1 t =0 to denote the momentum factors when using the decoupled momentum. We use { ( θ S f,t , θ S r,t ) } T -1 t =0 , { ( θ D f,t , θ D r,t ) } T -1 t =0 to represent the corresponding updated parameters. We use α ∈ [0 , 1] to denote the momentum factor and η as the learning rate, then the update schemes for a shared and decoupled momentum factors are shown as follows:

Here, we use { ( ̂ g S f,i , ̂ g S r,i ) } T -1 i =1 and { ( ̂ g D f,i , ̂ g D r,i ) } T -1 i =1 to denote the stochastic gradients during unlearning in the case of shared momentum and the decoupled momentum, respectively. Based on Algorithm 1, we update the parameters by gradients from the forget loss and the retain loss alternately. Therefore, we have g S f,i = ∇ θ L f ( θ S r,i -1 ) , g D f,i = ∇ θ L f ( θ D r,i -1 ) , g S r,i = ∇ θ L r ( θ S f,i ) , g D r,i = ∇ θ L r ( θ D f,i ) , and ̂ g S f,i , ̂ g D f,i , ̂ g S r,i , ̂ g D r,i are their stochastic variants.

When using decoupled momentum factors, the momentum factors { m D f, 0 , m D f, 1 , ..., m D f,T -1 } , { m D r, 0 , m D r, 1 , ..., m D r,T -1 } are two independent sequences. By contrast, when using a shared momentum factor, the factor is updated as a sequence { m S f, 0 , m S r, 0 , m S f, 1 , m S r, 1 , ..., m S f,T -1 , m S r,T -1 } . When initialization, we let θ S r, -1 = θ D r, -1 = θ o and m S r, -1 = m D f, -1 = m D r, -1 = 0 .

We focus on the variance of the parameters in two different update schemes in (5), both of which iteratively calculate the gradients on random variables. Therefore, we first estimate the variance caused by gradient calculation as in the following lemma.

Lemma 3.4. (Variance of Gradients) If the loss function L is Lipschitz smooth with a constant L , and Var( θ ) ≤ σ 2 θ , then we have Var( ∇ θ L ( θ )) ≤ L 2 σ 2 θ .

The proof and discussions are deferred to Appendix A.1. We can use Lemma 3.4 to derive the maximum variance of model parameters for both update schemes in (5) as in following theorem.

Theorem 3.5. (Variance Bound Comparison for Decoupled vs. Shared Momentum) For the update schemes indicated in (5) using the same hyperparameters ( η , α ), and we use Var( · ) to denote the maximum variance of a variable, if the function L f , L r and the stochastic gradient { ( ̂ g S f,i , ̂ g S r,i ) } T -1 i =0 , { ( ̂ g D f,i , ̂ g D r,i ) } T -1 i =0 satisfy Assumption 3.1, 3.2, and 3.3, then

The proof is deferred to Appendix A.2. Theorem 3.5 formalizes that decoupled momentum induces smaller worst-case parameter variance compared to shared momentum, demonstrating that decoupled momentum theoretically enhances the stability of unlearning. Additionally, we prove that decoupled momentum can also induce smaller worst-case variance of downstream metrics in Appendix A.3.

3.4 Dual Optimizers for Machine Unlearning

Algorithm 1 Machine Unlearning with Shared Optimizer / Dual Optimizers 1: Input: Model: f θ ; Forget set: D f ; Retain set: D r ; Iterations for outer loop: T o ; Iterations for forgetting: T f ; Iterations for retaining: T r ; Step sizes: η , η f , η r . 2: Optim is the same optimizer as in pretraining with step size η . Optim f is Adam( θ, η f ) , Optim r is the same optimizer as in pretraining with step size η r . 3: for t = 1 , ..., T o do 4: for t ′ = 1 , ..., T f do 5: Fetch mini-batch data from the forget set B f ∼ D f 6: Calculate the forget loss L f on B f and get the gradient 7: Use Optim / Optim f to update θ 8: end for 9: for t ′ = 1 , ..., T r do 10: Fetch mini-batch data from the retain set B r ∼ D r 11: Calculate the retain loss L r on B r and get the gradient 12: Use Optim / Optim r to update θ 13: end for 14: end for 15: Output: Model f θ

We incorporate the findings of the two subsections above and propose Dual Optimizers ( DualOptim ) to enhance the efficacy and stability of MU. Specifically, we utilize two distinct optimizers during the unlearning process: Adam for forgetting and the default optimizer (e.g., SGD) for retaining. On one hand, we use the same optimizer as in pretraining to maintain the performance on the retain set and use the optimizer with adaptive learning rates to handle the large gradient variation for the forget loss. On the other hand, we decouple the momentum factors and use two distinct optimizers to boost the algorithm stability during unlearning. Notably, DualOptim is a plug-and-play approach and can be integrated in existing MU algorithms that minimize forgetting and retaining objectives alternately. The detailed pseudo-code to compare the pipeline of MU with a shared optimizer and DualOptim is presented in Algorithm 1, with unique segments for shared and dual optimizers highlighted in red and green, respectively. Algorithm 1 presents a general framework, L f and L r are specified by different concrete MU algorithms.

4 Experiments

In this section, we conduct extensive experiments to evaluate our method for different applications, including image classification, image generation, and large language models. The results demonstrate that our method can enhance the efficacy and stability of multiple MU methods, achieving new stateof-the-art performance across diverse scenarios. We conduct ablation studies for further analysis.

4.1 Random Subset Unlearning in Image Classification

Table 1: Performance summary of MU methods for image classification. Experiments are conducted on (a) 10% random subset of CIFAR-10 using ResNet-18 and (b) 10% random subset of TinyImageNet using Swin-T . All results are presented as mean and standard deviation across 5 trials with different random forget data. Performance gaps with RT are indicated in blue. The average gap ( Gap ) and average standard deviation ( Std ) metrics are calculated by the average of the gaps and standard deviation measured in FA, RA, TA, and MIA, respectively. All the numbers are in percentage.

We start with random subset unlearning tasks in image classification, including using ResNet-18 [45] on CIFAR-10 [46] and Swin Transformer-Tiny (Swin-T) [47] on TinyImageNet [48]. Additional results on CIFAR-100 [46] and SVHN [49] are included in Appendix C.3. Consistent with previous work [21, 24, 6, 7], we regard Retrain (RT) as the gold standard of MU. Since our proposed DualOptim is plug-and-play, we apply it to SCRUB [8], SalUn [6], and SFRon [7] to validate its efficacy and stability. Additionally, we include three simple baselines, namely Fine-tune (FT) [50], Gradient Ascent (GA) [21] and Random Label (RL) [22], for reference as well. Note that the hyperparameters of all evaluated methods are tuned to their optimal values by sophisticated search, and we defer the implementation details to Appendix B. Besides the metrics mentioned in Sec. 3.1, which includes FA, RA, TA and MIA, we follow the evaluation criteria in [6, 7] to report the average gap (Gap) and average standard deviation (Std) to indicate the overall performance and stability, respectively. They are the average gap with RT and the average standard deviation among the results in FA, RA, TA, and MIA, respectively.

As illustrated in Table 1, SFRon achieves the smallest average gap with RT, yet it exhibits the highest variability in performance across different random forget subsets. This suggests that the optimal hyperparameters vary significantly depending on the specific forget sets used. Despite being more stable, other methods yield suboptimal results in terms of unlearning efficacy. By integrating DualOptim into these MU algorithms, we observe notable enhancements in terms of both unlearning efficacy and performance stability. For example, in the task of unlearning a 10% random subset from CIFAR-10, DualOptim reduces the average standard deviation of SFRon from 2.33% to 0.66%, while narrowing the average gap from 0.51% to 0.44%, achieving the best performance among all. Additionally, DualOptim significantly narrows down the average gap with RT for both SCRUB and SalUn, with improvements of 1.45% and 2.54%, respectively. Consistent results are observed in other datasets and model architectures as well, highlighting the widespread effectiveness of DualOptim in improving both the performance and stability of unlearning processes.

4.2 Class-wise Unlearning in Image Generation

Besides classification, we further evaluate our proposed DualOptim in class-wise unlearning tasks in image generation. Following the settings of [7], we conduct experiments using conditional DDPM [26] with the U-Net [51] on CIFAR-10 and the latent diffusion model [28] with Diffusion Transformer (DiT) [29] on ImageNet [48]. We apply our proposed DualOptim to SFRon [7], which exhibit the state-of-the-art unlearning performance in image generation. We also include SA [35] and SalUn [6] as baselines for comparison. Note that we do not apply DualOptim to SA and SalUn due to their joint updating scheme and unstable performance on ImageNet. Nonetheless, we still present the results of SalUn+DO on CIFAR-10 in Appendix C.4. The implementation details can be found in Appendix B. As for the evaluation metrics, we adopt the accuracy of the unlearned models generated images on forgetting classes (FA) by a pre-trained classifier to indicate the forgetting efficacy and Fréchet Inception Distance (FID) [52] to assess the generation capability on the retained classes.

As shown in Table 2 and 3, our method generally enhances the fidelity of generated images for retained classes while ensuring high forgetting efficacy and low variance. For example, it significantly reduces the FID of SFRon for the white wolf category from 23.28 to 14.63, with FA remaining at 0.00%. Importantly, these results demonstrate that DualOptim is effective across various datasets and model architectures, making it suitable for practical applications. To-

gether with the findings in Section 4.1, these results highlight the effectiveness and generalizability of DualOptim in diverse computer vision tasks. Generated images from the unlearned model utilizing DualOptim can be found in Appendix C.7. The visualization again indicates that DualOptim achieves effective unlearning while maintaining the generation capability for the retained classes.

4.3 Random Subset Unlearning in Large Language Models

Following the success of MU in vision tasks, there is a rising interest of applying it to remove private or harmful information from large language models (LLMs). We conduct the experiments on Phi-1.5 [53] and LLaMA 2 [37], both are fine-tuned on TOFU dataset [2] with 1.3B and 7B parameters, respectively. We include three untargeted unlearning methods (GA+GD, NPO+GD and ME+GD) and two targeted unlearning methods (DPO+GD and IDK+AP) [9] as the baselines. We employ

DualOptim (DO) in ME+GD and IDK+AP, which perform the best in [9], to validate the effectiveness of our method. Consistent with [2, 9], we consider three levels of unlearning tasks: to forget 1%, 5%, and 10% of the constructed data. We follow [9] and adopt the improved Model Capability (MC) 3 and Forger Efficacy (FE) as the evaluation metrics.

The results in Table 4 indicate that our method significantly enhances both FE and MC in general. Although a slight reduction in MC is observed for some instances, the more effective forgetting achieved ultimately leads to an increase in the average metric for all cases. Note that the standard deviations are omitted since they are smaller than 5% of the corresponding mean values in all cases. In addition, as shown in Figure 7 of Appendix C.5, the unlearning process utilizing DualOptim demonstrates greater effectiveness and stability compared wtih other baselines. These findings underscore the effectiveness of our method in terms of both performance and stability for LLMs.

It is important to note that the forgetting and retaining objectives are jointly optimized in the baselines listed in Table 4, whereas our method alternates between optimizing these two objectives. To ensure a fair comparison, we also evaluate unlearning performance by alternately optimizing the forgetting and retaining objectives using a shared optimizer. The results in Table 15 of Appendix C.5 demonstrate that DualOptim surpasses both the joint and alternate update schemes.

Table 5: Running time and memory usage of Adam and DualOptim on Llama 2 with full-parameter tuning and LoRA. Note that for full-parameter tuning, we used two H20 GPUs and DeepSpeed zero stage 2 is adopted; for LoRA, we used a single H20 GPU.

(a)

Running Time Usage (min)

(b)

Memory Usage (GB/GPU)

Although DualOptim exhibits effectiveness in MU for LLMs, as shown in Table 5, the decoupled momentum introduces approximately 1 . 5 × computational and memory overhead because of large amount of parameters in LLMs. To investigate our method's effectiveness under limited computation

3 Model Capability (MC) is referred to Model Utility (MU) in [9], which conflicts with the abbreviation of Machine Unlearning (MU) in this paper.

resources, we apply DualOptim to the parameter-efficient fine-tuning techniques, such as LoRA [54]. As indicated in Table 16 of Appendix C.5, DualOptim achieves a minimal compromise in unlearning performance by using LoRA, while significantly reducing memory consumption on optimizer states.

4.4 Ablation Studies

In this subsection, we compare different combinations of optimizers to further analyze the proposed method. More ablation studies can be found in Appendix C.6.

We compare various combinations of the Adam and SGD optimizers for the forget loss and the retain loss with their standalone counterparts. Besides Adam, we also investigate the combinations of SGD and other optimizers, e.g., Lion [55] and Muon [56]. The results in Table 6 indicate that the configuration of Adam (F) + SGD (R) offers the optimal balance between performance and stability. This finding underscores the importance of decoupled momentum and adaptive learning rate for the forgetting objective. Moreover, the results presented in Table 18 and Table 19 in Appendix C.6 indicate that the best optimizer for retaining depends on the choice of optimizer during pretraining and the specific MU algorithms employed. It should be pointed out that a shared SGD demonstrates a competitive average performance but suffers from high standard deviation indicating instability. Conversely, Lion exhibits low variability but does not achieve optimal performance. These insights further emphasize the efficacy of our method in enhancing performance and stability in MU tasks.

5 Conclusion

This study improves efficacy and stability in approximate machine unlearning (MU) methods with Dual Optimizers (DualOptim). By integrating adaptive learning rates and decoupled momentum, DualOptim enhances unlearning performance across diverse applications in computer vision and natural language processing. Its plug-and-play design ensures easy integration into existing MU frameworks. DualOptim marks a significant advancement in the MU field, offering an effective solution to meet the demand for trustworthy machine learning systems.

Broader Impacts and Limitations

Our method contributes to the trustworthiness of deep learning and can be broadly applicable across diverse tasks. Although DualOptim can be integrated with parameter-efficient fine-tuning methods, it still introduces double memory consumption for optimizer states. We leave the development of a more efficient approach as future work.

Acknowledgments and Disclosure of Funding

This work is supported by National Natural Science Foundation of China (NSFC Project No. 62306250) and City University of Hong Kong (CityU Project No. 9220132).

References

Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer. Red-teaming the stable diffusion safety filter. In NeurIPS ML Safety Workshop , 2022.

Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. In First Conference on Language Modeling , 2024.

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning , pages 3832-3842. PMLR, 2020.

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics , pages 2008-2016. PMLR, 2021.

Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11516-11524, 2021.

Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations , 2024.

Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient-based machine unlearning with remain geometry enhancement. Advances in Neural Information Processing Systems , 37:26377-26414, 2024.

Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems , 36:1957-1987, 2023.

Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. In International Conference on Learning Representations , 2015.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP) , pages 141-159. IEEE, 2021.

Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape of machine unlearning: A comprehensive survey and taxonomy. IEEE Transactions on Neural Networks and Learning Systems , 2024.

Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence , 2024.

Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory , pages 931-962. PMLR, 2021.

Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems , 34:18075-18086, 2021.

Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems , 32, 2019.

Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory , pages 4126-4142. PMLR, 2021.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR, 2017.

Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. A swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1139-1147. PMLR, 2019.

Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) , pages 303-319. IEEE, 2022.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9304-9312, 2020.

Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems , 36:51584-51605, 2023.

Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 7210-7217, 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4195-4205, 2023.

Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22522-22531, 2023.

Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23) , pages 5253-5270, 2023.

Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2426-2436, 2023.

Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22691-22702, 2023.

Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-menot: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1755-1764, 2024.

Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems , 36:17170-17194, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 2038-2047, 2022.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations , 2023.

Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations , 2024.

Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. Avoiding copyright infringement via machine unlearning. arXiv e-prints , pages arXiv-2406, 2024.

Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence , pages 1-14, 2025.

Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling , 2024.

Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2615-2632, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2011, page 4. Granada, 2011.

Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. Annual Network and Distributed System Security Symposium , 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 234-241. Springer, 2015.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 , 2023.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems , 36:49205-49233, 2023.

Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.

A Proofs

A.1 Proof of Lemma 3.4

Proof. Let θ 1 , θ 2 be independent copies of θ with probability density p ( θ ) . Using the variance identity Var( X ) = 1 2 E [ ‖ X 1 -X 2 ‖ 2 ] for independent copies X 1 , X 2 , we have:

By L -smoothness, ‖∇L ( θ 1 ) -∇L ( θ 2 ) ‖ ≤ L ‖ θ 1 -θ 2 ‖ , we have:

Since Var( θ ) ≤ σ 2 θ , we have 1 2 ∫∫ p ( θ 1 ) p ( θ 2 ) ‖ θ 1 -θ 2 ‖ 2 dθ 1 dθ 2 ≤ σ 2 θ . Therefore, we can conclude Var( ∇ θ L ( θ )) ≤ L 2 σ 2 θ

Discussion. Lemma 3.4 is an important tool for us to analyze the variance of the full-batch gradient, because the full-batch gradient is calculated based on the model parameters obtained in the last iteration, which is also a random variable. Specifically, we can derive the following inequality from the update rule in (5) for further analysis. We add no superscripts, indicating it is applicable for both update schemes.

A.2 Proof of Theorem 3.5

Proof. Without the loss of generality, we can assume the pretrained parameter θ o = 0 and the learning rate η = 1 , because shift and multiplication do not change the inequalities in the conclusion to prove.

Unfold the update rules in (5) and we obtain

For notation simplicity, we let A k = ∑ k i =0 α i . It is clear that ∀ k 1 ≤ k 2 , A k 1 ≤ A k 2 . Based on the update scheme (5) and η = 1 , we have:

Combining (10) and (11), we have the following equations. For notation simplicity, we let A k = ∑ k i =0 α i . It is clear that ∀ k 1 ≤ k 2 , A k 1 ≤ A k 2 .

We prove the theorem by mathematical induction.

We start with the case of t = 0 . Based on (12), we have θ S f, 0 = -̂ g S f, 0 , θ D f, 0 = -̂ g D f, 0 . Both are stochastic gradients calculated on the initial parameter θ o , so based on Assumption 3.1, we have:

In addition, we have θ D r, 0 = -̂ g D f, 0 -̂ g D r, 0 and θ S r, 0 = -(1 + α ) ̂ g S f, 0 -̂ g S r, 0 based on (12). Based on Lemma 3.4 and inequality (9), we have:

We assume negative correlation between gradients from the forget loss and the retain loss in Assumption 3.2. Therefore, we can derive the variance for θ S r, 0 and θ D r, 0 as follows:

Thus, we have the following conclusion:

That is to say, the conclusion of the theorem holds for t = 0 . Now we assume that the conclusion of the theorem holds for 0 , 1 , ..., t -1 , and then consider the case of t .

Based on Assumption 3.2, we have the following inequality for both update schemes:

glyph[negationslash]

glyph[negationslash]

Similarly, we have ∀ w 0 , ..., w t , Var ( ∑ t i =0 w i ̂ g r,i ) ≤ (1 + tτ ) ∑ t i =0 w 2 i Var( ̂ g r,i ) .

Now we combine (12), (14), (17) and can derive the following inequalities:

By definition, Var( θ D f,t ) , Var( θ S f,t ) , Var( θ D r,t ) , and Var( θ S r,t ) are the maximum possible value of Var( θ D f,t ) , Var( θ S f,t ) , Var( θ D r,t ) , and Var( θ S r,t ) , respectively. Considering the inequalities in (18) are all achievable, we have the following:

By induction, we have for t ′ ∈ { 0 , 1 , ..., t -1 } , Var( θ D f,t ′ ) ≤ Var( θ S f,t ′ ) , Var( θ D r,t ′ ) ≤ Var( θ S r,t ′ ) . In addition, ∀ k 1 ≤ k 2 , A k 1 ≤ A k 2 . Therefore, after comparing the factors and terms on the right hand of (19), it is obvious to obtain the following conclusion.

By induction, the conclusion of this theorem holds for any t ≥ 0 .

A.3 Variance Bound of Performance Metric Function

We let M ( θ ) to represent the performance of model with parameter θ where M can be FA/RA/TA/MIA. Before we derive the variance bound of performance metric function M ( θ ) , we make the following assumption:

Assumption A.1. The performance metric function M is Lipschitz continuous:

where L M is a non-negative constant.

Following a similar logic to Lemma 3.4, we have the following corollary:

Corollary A.2. If Assumption A.1 holds, and the parameter variance satisfies Var( θ ) ≤ Var( θ ) , then we have Var( M ( θ )) ≤ L 2 M Var( θ ) .

Proof. Similar to the proof of Lemma 3.4, we have:

where θ 1 , θ 2 are independent copies of θ with probability density p ( θ ) .

By ‖ M ( θ 1 ) -M ( θ 2 ) ‖ ≤ L M ‖ θ 1 -θ 2 ‖ , we have:

Since Var( θ ) ≤ Var( θ ) , we have 1 2 ∫∫ p ( θ 1 ) p ( θ 2 ) ‖ θ 1 -θ 2 ‖ 2 dθ 1 dθ 2 ≤ Var( θ ) . Therefore, we can conclude Var( M ( θ )) ≤ L 2 M Var( θ ) .

Discussion. The final unlearned parameters are usually the parameter updated by retaining loss at the last iteration T , i.e., θ D r,T using decoupled momentum and θ S r,T using shared momentum. According to Theorem 3.5, we have Var( θ D r,T ) ≤ Var( θ S r,T ) . Let the upper bound of the performance variance Var( M ( θ )) = L 2 M Var( θ ) , we can derive Var( M ( θ D r,T )) ≤ Var( M ( θ S r,T )) . Thus, decoupled momentum can also induce less variability on performance metrics .

B Implementation Details

B.1 Implementation Details for Image Classification

For CIFAR-10 , CIFAR-100 , and SVHN using ResNet-18 , all baselines use the SGD optimizer with momentum of 0 . 9 , weight decay of 5 × 10 -4 , and batch size of 128 if not specified. For TinyImageNet , Swin-T , the models are initialized from torchvision weight pre-trained on ImageNet. All baselines use the Adam optimizer with a weight decay of 5 × 10 -4 and batch size of 128 if not specified. Summaries of the hyperparameters for each method on each dataset are shown in Table 7 - 10. Note that the unspecified hyperparameters are the same as the default ones reported in their original papers.

B.2 Implementation Details for Image Generation

For CIFAR-10 , we use DDPM based on U-Net architecture with 1000 timesteps for linear β schedule. All methods use Adam optimizer and batch size of 128 . The hyperparameters of SalUn+DO are: T out = 150 , T in = 1 , η f = 4 × 10 -5 , η r = 1 × 10 -4 , threshold = top50% . The hyperparameters of SFRon+DO are: T out = 150 , T in = 1 , η f = 1 × 10 -4 , η r = 1 × 10 -4 , α = 1 , λ = 0 . 5 , γ = 3 . The hyperparameters of other methods are the same as those reported in [7].

For ImageNet , we use pre-trained DiT-XL /22 with 256 × 256 resolution. All methods use AdamW optimizer and batch size of 1 . The hyperparameters of SFRon+DO are: T out = 500 , T in = 1 , η f = 5 × 10 -5 , η r = 5 × 10 -5 , α = 1 , γ = 3 . Since the batch size is 1, we ignore λ in adaptive coefficients. The hyperparameters of other methods are the same as those reported in [7].

B.3 Implementation Details for Natural Language Processing

For Phi-1.5 and LLaMA 2 , we utilize pre-trained models on the TOFU dataset [2] and conduct evaluations accordingly. For the baseline methods, including GA+GD and DPO+GD , as well as the methods ME+GD and IDK+AP , we adopt the default hyperparameters reported in [9]. For IDK+AP , the forget coefficient and regularization coefficient are set to 1 . 0 across all models. For ME+GD , the retain coefficient is 1 . 0 for all methods. The forget coefficient is 1 . 0 for LLaMA 2, while it is set to 0 . 1 for LLaMA 2 LoRA and Phi 1.5. For the LLaMA LoRA configuration, the LoRA rank is 8 and the LoRA alpha is 32. For training LLaMA 2 with DualOptim, we use two NVIDIA H20 GPUs with 96GB of memory each. For Phi-1.5, we use two NVIDIA RTX 6000 Ada GPUs with 48GB of memory each. The detailed hyperparameters of our methods are reported in Table 12.

C Additional Results

C.1 Non-positive Correlation between Forget and Retain Gradients

Figure 4 illustrates non-positive correlation between ̂ g f and ̂ g r during unlearning, supporting the reasonableness of Assumption 3.2. Note that the results are obtained using SGD optimizer.

C.2 Unlearning Process of Other MU Methods

The unlearning processes of SalUn [6] and SCRUB [8] are illustrated in Figure 5 and 6, respectively. Compared to SFRon (see in Figure 2), SalUn and SCRUB exhibits less performance fluctuation. However, they underperform SFRon by a large margin. Despite that, our method is still effective in boosting their unlearning performance.

0

200

400

600

800

1000 1200 1400

Iteration

0.12

0.10

0.08

0.06

0.04

0.02

0.00

0.02

cos

(

gf

,

gr

)

0

2

4

6

8

10

Iteration

94

95

96

97

98

99

FA

SGD

Adam

DualOptim

Retrain

(a) Forget Accuracy (FA)

0

2

4

6

8

10

Iteration

98.8

99.0

99.2

99.4

99.6

99.8

100.0

RA

SGD

Adam

DualOptim

Retrain

0

2

4

6

8

10

Iteration

93.0

93.5

94.0

94.5

TA

SGD

Adam

DualOptim

Retrain

(c) Test Accuracy (TA)

0

2

4

6

8

10

Iteration

60

65

70

75

80

85

90

95

MIA

SGD

Adam

DualOptim

Retrain

(d) MIA

0

2

4

6

8

10

Iteration

93

94

95

96

97

98

99

FA

SGD

Adam

DualOptim

Retrain

(a) Forget Accuracy (FA)

0

2

4

6

8

10

Iteration

98.6

98.8

99.0

99.2

99.4

99.6

99.8

100.0

RA

SGD

Adam

DualOptim

Retrain

0

2

4

6

8

10

Iteration

93.00

93.25

93.50

93.75

94.00

94.25

94.50

94.75

TA

SGD

Adam

DualOptim

Retrain

(c) Test Accuracy (TA)

0

2

4

6

8

10

Iteration

75.0

77.5

80.0

82.5

85.0

87.5

90.0

92.5

MIA

SGD

Adam

DualOptim

Retrain

(d) MIA

C.3 Additional Results of Image Classification

We further conduct experiment to evaluate the performance of unlearning 50% random subsets of CIFAR-10 and 10% random subsets of CIFAR-100 and SVHN. The results are reported in Table 13. The observation in Table 13 is consistent with that in Table 1, emphasizing the efficacy of our method across different fractions of forget data, different datasets and different model architectures.

(a) CIFAR-10 Random Subset Unlearning (50%)

(b) CIFAR-100 Random Subset Unlearning (10%)

(c) SVHN Random Subset Unlearning (10%)

C.4 Additional Results of Image Generation

To further validate the effectiveness of the proposed method, we apply DualOptim to SalUn [6] on CIFAR-10 with DDPM. Table 14 illustrates that DualOptim can still enhance the performance and stability of SalUn in image generation tasks. Note that we adapt SalUn to alternate updating scheme when applying DualOptim, which originally utilizes joint updating scheme in image generation tasks. We do not include its results on ImageNet due to suboptimal and unstable performance.

C.5 Additional Results of Large Language Models

The results in Table 15 suggest that the alternate updating method can improve LLM unlearning. In addition, DualOptim boosts the performance based on that. As presented in Table 16, DualOptim achieves a minimal compromise in unlearning performance by using LoRA, while significantly reducing memory consumption. Furthermore, Figure 7 illustrates that the unlearning process utilizing DualOptim demonstrates greater effectiveness and stability compared to other baselines.

Table 15: Performance comparison of different updating methods on TOFU-finetuned Phi-1.5 . The MU method is IDK+AP . The results include Model Capability (MC), Forget Efficacy (FE), and the average metric (Avg.) for forgetting 1%, 5%, and 10% data. Note that Joint represents jointly optimizing forgetting and retaining objectives, which is the default updating method; Alternate represents alternately optimizing these two objectives using a shared optimizer.

C.6 Additional Ablation Studies

Decouple m and v in Adam. Given that Adam incorporates two momentum terms, i.e., the first moment m and the second moment v , we conducted a further analysis to explore the impact of decoupling these terms. As shown in Table 17, decoupling both m and v yields the best performance. This finding reinforces the importance of fully decoupling the optimization processes for forgetting and retaining objectives.

Ablation studies on Adam-trained ResNet-18 and SCRUB [8]. We conduct ablation studies based on the configurations of Adam-trained ResNet-18 + SFRon and SGD-trained ResNet-18 + SCRUB. As illustrated in Table 18 and 19, Adam (F) + Adam (R) is the best configuration in both cases, indicating that the optimal optimizer for retaining is influenced by the choice of optimizer during pretraining and the specific MU algorithms employed.

Shared optimizer with larger momentum coefficient. We further compare DualOptim with shared optimizers that employ larger momentum coefficients. As observed in Table 20, while increasing

0.0

0.2

0.4

0.6

0.8

0.3

0.4

0.5

0.6

0.7

0.8

forget

1% data

0.0

0.2

0.4

0.6

0.8

forget

5% data

0.0

0.2

0.4

0.6

0.8

forget

10

% data

bili

ty

Forget Efficacy

GA+GD

ME+GD

ME+GD+DO

(a) untargeted unlearning

0.0

0.2

0.4

0.6

0.8

0.3

0.4

0.5

0.6

0.7

0.8

forget

1

% data

0.0

0.2

0.4

0.6

0.8

forget

5

% data

0.0

0.2

0.4

0.6

0.8

forget

10

% data

bili

ty

Forget Efficacy

DPO+GD

IDK+AP

IDK+AP+DO

(b) targeted unlearning

the momentum coefficient α can reduce performance variation when using SGD, it also results in slower convergence and ultimately leads to suboptimal performance. In contrast, increasing β 1 has only a marginal impact when using Adam. This is because Adam's momentum update rule, i.e. m = β 1 m + (1 -β 1 ) ̂ g , inherently provides a stronger smoothing effect compared to SGD, i.e. m = α m + ̂ g , when the same momentum coefficient is applied.

C.7 Visualization for Machine Unlearning in Image Generation

Generated images from the unlearned model utilizing DualOptim are shown in Figure 8 and 9. The visualization indicates that, by leveraging DualOptim, effective unlearning is achieved and the generation capability for the remaining classes is retained.

(i) Forgetting Ship

(j) Forgetting Truck

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This paper discusses the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theoretical result, the paper provides the full set of assumptions and a complete proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper discloses all the information needed for reproduction.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: This paper provides open access to the code in supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper specifies all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This paper report the standard deviation of results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This paper provides sufficient information on the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses broader impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper utilizes public pretrained language models and image generators and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets, used in the paper, are properly credited. The license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a “dual-optimizer” strategy for approximate machine unlearning (MU). During unlearning, gradients from the forget loss are updated with Adam, while gradients from the retain loss continue with the model’s original optimizer (typically SGD); their momentum buffers are fully decoupled. The authors prove a tighter bound on parameter-variance under decoupled momentum and demonstrate consistent gains and markedly lower run-to-run variance across MU on image classifiers, class-wise unlearning in generative diffusion model, and 7 B-parameter LLM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The idea is simple yet effective, it requires minimal code change to deploy in the real world.\n- The authors provides very comprehensive empirical validation with convincing details on implementation and experimental setup. Open sourcing the code and results will help the unlearning community greatly. \n- The theoretical analysis provides a clear variance bound links the decoupled-momentum idea to stability, providing more than just empirical intuition.\n\nWeaknesses\n- Assumption 3.2 treats forget-retain gradient correlation as negligible, yet Figure 4’s cosine-similarity plot is not a rigorous test of that claim. Firstly, from intuition, forgetting objective and retaining objective is possibly highly negatively correlated during a whole training session (across iterative updates). Secondly, cosine similarity treats the *coordinates* of a gradient vector as “samples” and asks whether the two vectors are roughly orthogonal **within a single timestep**. The correlation in Assumption 3.2, by contrast, is about how two *random variables* (the gradient estimates produced by data sampling) co-vary **across different draws**. These are different statistical objects. Also, due to curse of dimensionality, the reported value in Figure 4 may simply reflect high dimensionality, not genuine near-zero correlation. \n- The paper’s overall idea is slightly too incremental, the impact at conceptual level is limited. (but this weakness is supplemented by the comprehensiveness of the empirical result, which I as a reviewer do appreciate.)\n- More advanced optimizers and their combinations could be included in the experimental/ablation study setup.\n\nQUESTIONS:\n- Could the authors quantify wall-clock overhead and GPU memory increase on the LLaMA-2 experiment?\n- How sensitive is DualOptim when the pre-training optimizer is already adaptive (e.g., AdamW, Adan)?\n- Typo in Section A.2 Proof of Theorem 3.5 Equation (11) Decoupled Momentum, $\\mathbf{m}^S_{f,i}$ should be $\\mathbf{m}^D_{f,i}$\n\nLIMITATIONS:\nMostly yes. However, it may be beneficial to include more discussion on the conditions under which DualOptim exhibits limited performance and the reasons behind it.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the instability that plagues many machine-unlearning algorithms by introducing DualOptim, a two-optimizer scheme: one adaptive optimizer for the forget objective and another for the retain objective, with their momentum terms fully decoupled. The authors supply a variance-bound analysis explaining why decoupled momentum should reduce parameter drift, and they validate the method on image-classification, image-generation, and large-language-model tasks. Across these settings, DualOptim consistently narrows the gap to full retraining and lowers variance, suggesting it is a practical “plug-and-play” upgrade for existing MU pipelines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. DualOptim effectively addresses the hyperparameter sensitivity issue prevalent in existing MU methods, providing significantly improved stability across diverse deployment scenarios.\n\n2. The approach demonstrates consistent performance gains across multiple domains including image classification, generation, and LLMs, showcasing its broad applicability.\n\n3. The solution combines both theoretical foundations (adaptive learning rates and decoupled momentum) and extensive empirical validation, ensuring reliable unlearning performance.\n\n**Weaknesses**\n\n1. Using different optimizers or decoupled momentum for multi-objective training is familiar from multitask and bilevel learning. The paper does not clearly separate its contribution from those earlier ideas.\n\n2. Theoretical assumptions like uncorrelated forget/retain gradients may not hold in practice, and the paper provides little guidance on when DualOptim could underperform.\n\n3. Maintaining two optimizers doubles state storage and may slow training, yet the main text offers no quantitative overhead analysis, especially for large models.\n\nQUESTIONS:\n1. Are there examples where decoupling the optimizers actually hurts convergence?\n\n2. Can you provide concrete measurements of memory usage and run-time overhead when scaling to large LLMs? How do these costs compare with single-optimizer baselines?\n\n3. Have you evaluated DualOptim against automatic hyper-parameter scheduling techniques designed to stabilize MU?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DualOptim, a novel method designed to enhance the efficacy and stability of approximate machine unlearning (MU) methods. DualOptim addresses the instability and hyperparameter sensitivity of existing MU techniques by employing two key innovations: 1. adaptive learning rates for the forgetting objective and 2. decoupled momentum updates for the forgetting and retaining objectives. The method is validated through extensive experiments across diverse tasks, including image classification, image generation, and LLMs. Results demonstrate that DualOptim significantly improves the performance and stability of existing MU algorithms, making it a versatile and impactful solution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is technically sound, with strong theoretical foundations supporting the proposed method. The decoupling of momentum is rigorously analyzed, and its impact on variance reduction is well-justified.\n2. Extensive experiments across multiple domains (image classification, image generation, and LLMs) demonstrate the generalizability of DualOptim.\n3. The paper is clearly written and structured, with detailed explanations of the methodology and experimental setup.\n\nWeaknesses\n1. The use of decoupled momentum introduces additional memory and computational overhead, particularly for large-scale models like LLMs. While the authors discuss this limitation, further optimization strategies could be explored.\n2. While the paper demonstrates the effectiveness of DualOptim, it lacks a detailed analysis of the sensitivity of its own hyperparameters (e.g., learning rates for forgetting and retaining objectives). Additionally, the experiments compare DualOptim with other methods using their default hyperparameters, which may not represent the optimal performance of these baselines. A fairer comparison would involve tuning all methods under the same conditions.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DualOptim, a method to resolve the instability and difficult tuning associated with existing machine unlearning (MU) methods. DualOptim employs two separate optimizers: an adaptive one, like Adam, for the volatile \"forget\" objective, and the original optimizer for the \"retain\" objective to preserve model performance. By decoupling the momentum between these two tasks, the method is shown to enhance the stability and efficacy of the unlearning process. Extensive experiments validate that DualOptim boosts performance across diverse applications, including image classification, image generation, and large language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. The idea is clear and intuitive. Using Adam for the “forget” objective and SGD for the “retain” objective is easy to implement and generally sensible.\n\n2. The experiments are comprehensive, spanning image classification, diffusion models and LLMs, showing the approach is not task-specific.\n\n3. Table 5 systematically varies the two optimizers and supports the chosen pairing (Adam-F + SGD-R)\n\n4. Authors acknowledge memory overhead in Broader-Impacts/Limitations, showing awareness of practical constraints\n\nWeaknesses:\n\n1. The entire proof is for plain SGD with momentum (η=1 is also hard-coded)  , but DualOptim actually uses Adam on the forget branch. Preconditioning fundamentally changes the variance dynamics, so the guarantees do not directly transfer.\n\n2. The paper's theory culminates in Theorem 3.5, which proves a smaller worst-case parameter variance for decoupled momentum. However, it does not mathematically connect this reduced variance to better performance on downstream metrics like the \"Gap\" to retraining or Membership Inference Attack (MIA) success.\n\n3. The paper states that all baselines were tuned via \"sophisticated search,\" but does not detail if the search budget was equal for the baselines versus DualOptim, which introduces two separate learning rates. his potential imbalance undermines the fairness of the comparison and could be the source of the reported stability gains.\n\n4. The main results compare DualOptim's alternating update schedule with baselines that sometimes use a joint optimization objective. While a fairer comparison is included in Appendix C.5, it is not featured in the main paper. This makes the primary results potentially misleading from a practical implementation standpoint.\n\nQUESTIONS:\n1. To quantify practical overhead, please report the wall-clock time and peak memory usage for DualOptim compared to a key baseline on a representative task.\n\n2. To validate your stability claims, please detail the hyperparameter tuning budget for both DualOptim and the baselines to ensure a fair comparison.\n\n3. Please address the mismatch between your theory (proven for SGD) and your algorithm (which uses Adam) by discussing the limitations and applicability of your proof.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}