Title: Unlearning-Aware Minimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Unlearning

Min-Max Optimization

Unlearning-Aware Minimization
Abstract: Machine unlearning aims to remove the influence of specific training samples (i.e., forget data) from a trained model while preserving its performance on the remaining samples (i.e., retain data). Existing approximate unlearning approaches, such as fine-tuning or negative gradient, often suffer from either insufficient forgetting or significant degradation on retain data.
In this paper, we introduce Unlearning-Aware Minimization (UAM), a novel min–max optimization framework for machine unlearning. UAM perturbs model parameters to maximize the forget loss and then leverages the corresponding gradients to minimize the retain loss. We derive an efficient optimization method for this min-max problem, which enables effective removal of forget data and uncovers better optima that conventional methods fail to reach.
Extensive experiments demonstrate that UAM outperforms existing methods across diverse benchmarks, including image classification datasets (CIFAR-10, CIFAR-100, TinyImageNet) and multiple-choice question-answering benchmarks for large language models (WMDP-Bio, WMDP-Cyber).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors focus on the machine unlearning problem, which aims to eliminate the influence of forget data while preserving the performance on the retain data. The article points out that solely using negative gradient or fine-tuning methods fail to effectively balance the objectives of maximizing forget loss and minimizing retain loss. And authors propose a novel min-max optimization framework to consistently achieve higher forget loss and lower retain loss. Based on first-order Taylor expansion, the paper proposes an efficient solution to the above optimization and theoretically analyzes the validity of the update strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The key innovation of this paper lies in formulating the machine unlearning problem within a min-max optimization framework, which leads to promising empirical performance. \n- By leveraging a first-order Taylor approximation, the authors transform a challenging bi-level optimization into a tractable update rule, enabling efficient gradient-based training. \n- The paper provides an intuitive geometric interpretation, illustrating how the alignment between the gradients of forget and retain data affects the optimization process, thereby enhancing the clarity and interpretability of the proposed method.\n\nWeakness:\n\nWhile the paper presents a novel and empirically effective framework for machine unlearning, several important aspects deserve further clarification and theoretical grounding:\n- About assumption and setting: The assumption that both the forget and retain data are known and accessible may not hold in realistic large-scale settings such as LLMs, where only the forget set is identifiable. This limits the method’s applicability in more practical scenarios.\n\n- About the technique: The method relies on an approximate optimization using a surrogate objective. However, it remains unclear whether the solution trajectory truly converges to the intended optimal parameter w^*,and how large the approximation gap is between the proxy and the true formulation. The clarification of the real gap is necessary for readers to understand the method.\n\n- About the theory for the effectiveness of model unlearning.\nThe effectiveness of forgetting (i.e., removing the influence of the forget set) and maintaining performance on the retain set is empirically verified, but lacks formal guarantees. Does the proposed objective ensure meaningful unlearning under general conditions? Is there a generalization bound or any guarantee of worst-case behavior?\n\nQUESTIONS:\n- In the related work 2.1, the authors mention that “our method is a min-max optimization framework rather than relying on dual-objective formulations.” What are the advantages of min-max optimization framework compared to dual-objective formulations?\n- From eq.3, it can be found that the variable in the minimization problem is w, which means the w* is fixed. And Eq.5 proposes a maximiaztion problem to estimate w*. Therefore, when we solve the maximization problem, the w* should be fix to w0+\\rho (\\nabla \\nabla L_f(w))/(||\\nabla \\nabla L_f(w)||_2^2). However, in the update process of eq.7, the w* is still treated as a variable. It seems the problem in eq.3 is not equal to that in eq.7.\n- Under random data setting, since the gradients calculated on retain data are similar to those calculated on forget data, it can be found that the results of UAM are relatively weaker, is there any solution to mitigate this problem?\n- In practical large-scale settings (e.g., LLMs), users may only provide forget sets. Is it realistic or necessary to assume full access to both retain and forget data partitions?\n- When using a first-order Taylor approximation, please specify the smoothness requirements for the function.\n- The graphical explanation in Figure 1 contains a typo: “GA”.\n- There is a grammatical error in line 113 on page 3: “where and Ω is a finite upper bound on the distance.”\n- The following sentence in the abstract may be ambiguous and could benefit from further clarification:“In this paper, we propose Unlearning-Aware Minimization (UAM), a novel min-max optimization framework that identifies weights with high loss on the forget data and uses their gradients to minimize the loss on the retain data.”\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulates machine unlearning as a minimax optimization problem. Specifically, it first maximizes the “forget” loss over a local delta-neighborhood of the parameters \nw, and then minimizes the “retain” loss based on that adversarial perturbation. Drawing inspiration from Sharpness-Aware Minimization, the inner maximization admits a closed-form approximation (Theorem 1), and the authors further leverage a first-order Taylor expansion for computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe manuscript is well structured and clearly written, with a thorough and precise overview of related work.\n\nWeaknesses\n\n1. It omits discussion of a closely related adversarial training approach to unlearning [1]. Reference [1] also considers a minimax optimization formulation to address unlearning problems. Could the authors provide further explanations on how their approach differs from [1]?\n\n2. The regularizer in Equation (2) compares losses at the optimal and approximate solutions but does not directly penalize the distance between them — why is that? Furthermore, the criterion for choosing $\\epsilon$ to ensure that $w^*$ lies within the $\\delta$-ball is unclear.\n\n3. The experimental comparisons do not include several recent baselines [2–5], particularly in Tables 1 and 3. Moreover, there should be an ablation study on the perturbation range $\\rho$.\n\n4. The procedure for generating the optimization trajectories shown in Figure 1 is not clear. Which model and data were used to obtain these trajectories?\n\nReferences\n\n[1] Discriminative Adversarial Unlearning\n\n[2] Large Language Model Unlearning via Embedding-Corrupted Prompts\n\n[3] MUSE: Machine Unlearning Six-Way Evaluation for Language Models\n\n[4] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning\n\n[5] Who's Harry Potter? Approximate Unlearning in LLMs\n\nQUESTIONS:\nNo more questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel unlearning framework, Unlearning-Aware Minimization (UAM). UAM effectively removes forget set information while preserving retain set information by min-max optimization which involves inner maximization and outer minimization.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength:\n- This paper proposes a novel unlearning method, Unlearning-Aware Minimization, which effectively remove forget data while maintaining general model performances.\n- Proposed methods are theoretically well described.\n- Proposed methods can be easily extended to other domains, such as multiple-choice question-answering using large language models.\n\n### Weakness:\n- Proposed methods introduce additional hyperparameters, such as γ and ρ, which needs to be tuned.\n- In image classification, comparison is limited to the early unlearning methods. Comparison with recent advanced unlearning methods will strengthen the effectiveness of UAM.\n\nQUESTIONS:\n- Although it showed superior unlearning performance compared to conventional methods, it seems not to work in a specific scenario where forget gradient is highly correlated with retain gradient. For example, in random data forgetting, if forget gradient from class 1 and retain gradient from class 1 may have same direction, then proposed objective function operates weirdly. Inferior unlearning performance in random data forgetting may be caused by this kind of reason. I want to ask the opinion of the authors about this concern.\n- Typos: In caption of Figure 1, gradient ascent (GA) is used instead of negative gradient (NG). In Table 1 and Table 5, some values in blue are not matched with the actual difference from Retrain. Some values even disturb interpreting overall results.\n\nLIMITATIONS:\n- UAM introduces additional hyperparameters, such as γ and ρ, in addition to the learning rate. It may need efforts to search optimal hyperparameters for each task, dataset, and model.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel machine unlearning framework called Unlearning-Aware Minimization (UAM) , which formulates unlearning as a min-max optimization problem. It identifies model weights within a local neighborhood that maximize the loss on the data to be forgotten (forget data) and then minimizes the loss on the remaining data (retain data) using gradients from these surrogate weights. It effectively removes the influence of the forget data while maintaining performance on the retain data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\nExperimental results show that UAM achieves lower accuracy on the forget data (indicating successful unlearning) while preserving high accuracy on the retain data, surpassing the considered methods such as FT, NG\n\nOn LLM-based unlearning tasks, UAM significantly reduces the model's ability to generate harmful or hazardous content while retaining strong general performance (e.g., MMLU scores).\n\nUAM does not rely on specific loss functions or architectures, allowing it to be applied broadly across domains including image classification and LLM unlearning..\n\n**Weaknesses**:\n\n1. The first-order Taylor approximation in Theorem 1 hinges on the assumption that \"$\\rho$ is sufficiently small.\" However, the paper lacks systematic validation of ρ’s impact across tasks. In high-dimensional or complex loss landscapes, this assumption may introduce non-negligible approximation errors, limiting the framework’s reliability.\n\n2. The critique of Fine-Tuning (FT) and Negative Gradient (NG) is thorough, but the analysis of Fisher Forgetting (FF) and Influence Unlearning (IU) is superficial. The paper underestimates challenges unique to these methods, weakening the comparative evaluation\n\n3. UAM is described as \"scalable and efficient\" but requires two gradient computations per iteration (inner/outer loops). Table 1 confirms the runtime of UAM exceeds FT and NG, contradicting efficiency claims and raising scalability concerns for larger models.\n\n4. The abstract claims UAM \"consistently outperforms existing methods,\" but in Table 3 (random forgetting), UAM exhibits higher $\\Delta Acc.$ and worse MIA-Eff than $l_1$-sparse. This contradicts the universality of the outperformance claim.\n\n5. In the experimental results comparison shown in Table 1, most of the compared methods are from 2022 or earlier, and there is a lack of comparisons with the latest state-of-the-art methods.\n\n6. Proofs for Lemma 1 and Theorem 1 are relegated to the Appendix. Given their foundational role in deriving UAM, these should appear in the main text to facilitate immediate verification.\n\n**Minor issues**:\n\n1. Line 129: \"sorely\" should be \"solely\".\n2. Line 140: \"an weight point\" should be \"a weight point\".\n\nQUESTIONS:\nSee my detailed comments.\n\nLIMITATIONS:\nSee my detailed comments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}