Title: Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Unlearning; Multi-objective optimization
Abstract: Machine unlearning (MU) aims to efficiently remove sensitive or harmful memory from a pre-trained model. The key challenge is to balance the potential tradeoff between unlearning efficacy and utility preservation, which involves forgetting undesirable information as defined while maintaining the model's original performance. One potential way to tackle this problem is to use multi-objective optimization to jointly optimize both the unlearning and utility preservation objectives. However, existing multi-objective methods only guarantee finding a Pareto-optimal solution without fine-grained control, which causes under-optimization of the unlearning objective. To this end, we first model MU as a constrained optimization problem, that is, optimizing the unlearning objective under the constraint of a bounded increase for utility loss. We then show that solving this optimization problem is equivalent to unilateral gradient surgery on the unlearning objective. To resolve the additional computational cost brought by gradient surgery, we propose an implicit gradient surgery method, which approximates the solution to the aforementioned constrained optimization problem via only one backpropagation, thereby achieving efficient utility-preserving MU. Theoretically, we provide a tight convergence analysis of the algorithm. Empirically, our extensive experiments show that the proposed algorithm achieves better tradeoff results than existing baselines. Codes are available at https://github.com/anseryuer/EUPMU-Efficient-Utility-Preserving-Machine-Unlearning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMachine unlearning aims to remove the memory of specific data from the pre-trained model, and there is a trade-off between increasing the effectiveness of unlearning and maintaining the performance of the original model. One way to mitigate this is to employ a multi-objective optimization, but existing methods are inadequate for unlearning. To address this issue, the paper formulates a constrained optimization problem that aims to optimize the unlearning objective under the constraint of a bounded increase for utility loss and proposes an efficient method based on gradient surgery to solve this problem.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. This paper accurately provides why multi-objective optimization cannot be directly applied to the challenges of machine unlearning.\n2. Each proposition is thoroughly proven in the appendix.\n3. The paper presents comprehensive experimental results, and the image generation outcomes clearly illustrate the effectiveness of the proposed method.\n\n## Weaknesses\n1. While existing unlearning studies [1, 2] are evaluated on datasets with diverse classes such as CIFAR-100, Tiny-ImageNet, and ImageNet-1k, the paper only conducts experiments on relatively simple datasets with 10 classes, such as CIFAR-10 and Imagenette. As a result, the evaluation appears insufficient to fully validate the generalizability and scalability of the proposed method.\n2. In Section 4.3, the definition of \"pure linearization\" is missing. There is no citation or clear description for \"RL\". Table 5 shows the proposed method takes more computation time than \"pure linearization\". If \"pure linearization\" refers to \"linear weighting,\" then it raises concerns about the effectiveness of the proposed method, which claims to achieve comparable computational efficiency to \"linear weighting\".\n3. While it is understandable that the hyperparameter \"error tolerance\" $\\varepsilon$ influences empirical performance, there is no sensitivity analysis of the performance to $\\varepsilon$. This lack of analysis makes it difficult to assess the robustness of the proposed method.\n4. Table 6 shows that the proposed method incurs a significantly higher computational cost compared to GA in the random data forgetting scenario, while achieving only marginal performance gains. This raises questions about its cost-effectiveness.\n5. While existing methods [1, 2] evaluate unlearning performance by quantifying the gap between an unlearned model and a model retrained from scratch without data subject to unlearning, this paper evaluates performance using raw accuracy values. Given that the objective of unlearning is to obtain a model equivalent to one trained without using the data, it would be more appropriate to evaluate performance based on the gap from the retrained model trained only on the retained data.\n\nMinor:\n\n6. Some highlights in Tables 5, 6, and 7 are incorrect.\n7. There is a typo in line 761 of Appendix F: \"$varepsilon_t$\" should be \"$\\varepsilon_t$\".\n\nReferences\n\n[1] Fan et al., Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation, In Proc. ICLR, 2024.\n\n[2] Jia et al., model sparsity can simplify machine unlearning, In Proc. NeurIPS, 2023.\n\nQUESTIONS:\n1. Could you clarify the precise definition of \"pure linearization\" used in Table 5 and provide a citation or detailed explanation of \"RL\" in Section 4.3?\n2. How sensitive is the performance to the value of the error tolerance $\\varepsilon$?\n3. Have you analyzed why the proposed method becomes less cost-effective specifically in the random forgetting scenario?\n4. Why is the evaluation process different from existing studies?\n\nLIMITATIONS:\nThe authors discussed the limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of balancing unlearning efficacy and utility preservation in machine unlearning (MU). The authors formulate the Utility Preserving Unlearning Problem as a constrained optimization: maximize the decrease in the unlearning loss under a bounded increase in the retaining loss. Empirical evaluations on image classification and generation tasks demonstrate that EUPMU outperforms baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. It is insightful that formulating MU as a constrained optimization problem with explicit control over utility loss.\n2. This paper provides detailed theoretical analysis and experimental evaluations that show consistent performance improvements.\n\nWeakness:\n\n1. Although it is very interesting to formulate MU as a gradient surgery problem, the gradient projection methods on MU have been investigated in the existing works [1][2]. I suggest that the authors add the corresponding discussions and comparison experiments.\n2. Some important methods are missed on the SD models, e.g., MACE [3] and UCE [4].\n3. The ablation experiments on the hyperparameters should also be added.\n4. Is it possible to implement EUPMU for multi-concept unlearning on the SD models?\n\n[1] Lin S, Zhang X, Susilo W, et al. GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 9087-9095.\n\n[2] Hoang T, Rana S, Gupta S, et al. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 4819-4828.\n\n[3] Lu S, Wang Z, Li L, et al. Mace: Mass concept erasure in diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6430-6440.\n\n[4] Gandikota R, Orgad H, Belinkov Y, et al. Unified concept editing in diffusion models[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 5111-5120.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the Utility-Preserving Unlearning Problem by formulating unlearning as a constrained optimization task. It proposes Unilateral Gradient Surgery to balance unlearning and utility preservation, and further introduces an efficient implicit variant to reduce computational cost. The authors provide theoretical analysis proving convergence to Pareto optimality, and experiments validate the method’s effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper is well-written and well-structured, with helpful illustrations that make the core ideas more accessible.\n\n2.The paper provides thorough theoretical analysis to support the proposed solution, which strengthens the soundness and credibility of the approach.\n\n3.The paper presents several practical unlearning scenario settings, which enhance the applicability of the proposed method and provide empirical evidence of its effectiveness.\n\nWeaknesses:\n\n1.The idea of leveraging gradient conflict mitigation for unlearning has been explored in prior work, which somewhat limits the novelty of this paper. Although the authors argue that their approach is motivated by a different perspective, I think this justification not entirely convincing. Additionally, the paper lacks discussion of several closely related works, such as GDR-GMA[1] and G-effect[2]. Notably, GDR-GMA also considers the unlearning process as a multi-task optimization problem.\n\n2.The paper lacks ablation studies on key hyperparameters, such as the error tolerance. Although the authors mention in the limitations section that related results are shown in Figure 7, I could not find any description in Figure 7 pertaining to error tolerance.\n\n3.The evaluation metrics used in this paper, at least for the classification tasks, raise some concerns and may not fully align with the goals of unlearning, which affects the overall assessment of the work. Specifically, treating lower UA and MIA scores and higher RA and TA scores as universally better may be misleading. Under such criteria, retraining might appear to perform poorly as an unlearning method. Moreover, overly aggressive unlearning might introduce new attack surfaces—for example, by exposing abnormal behavior on forgotten data—which highlights the necessity of maintaining a certain level of generalization. A rigorous unlearning approach should aim to closely match the behavior of retraining, rather than simply minimizing or maximizing certain metrics. \n\n[1]Lin, Shen, et al. \"GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n[2]Wang, Qizhou, et al. \"Rethinking llm unlearning objectives: A gradient perspective and go beyond.\" arXiv preprint arXiv:2502.19301 (2025).\n\nQUESTIONS:\n1.\tFurther clarifying the contributions of this paper, particularly in comparison to existing gradient conflict mitigation approaches. The current explanation that the difference lies in the principle origin is not entirely convincing. It would be helpful to elaborate on this distinction more clearly, ideally within the main body of the paper, as this would better highlight the novelty of the proposed method for the reader.\n\n2.\tThe issues related to the evaluation metrics should be revisited and addressed appropriately.\n\n3.\tIt would be helpful to further enhance the experimental section by providing additional results and including ablation studies on key hyperparameters.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to addresses the challenge of utility-preserving in machine unlearning. It first identifies the limitations of the conventional linearization approach, which suffers from utility loss, and the multi-objective optimization approach, which leads to insufficient unlearning and efficiency issues. To overcome these, the authors propose an Implicit Gradient Surgery method. This method features utility-preserving characteristics and guarantees sufficient optimization of the forgetting objective. Furthermore, an approximation technique within the method ensures computational efficiency comparable to the linear weighting approach. The paper also provides a Pareto convergence analysis, proving that the proposed algorithm achieves the aforementioned design goals. The experimental results robustly validate this theoretical finding.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The problem formulation and motivation are presented with clarity. Figure 1 illustrates the limitations of both linearization and multi-objective optimization methods in machine unlearning scenarios. The background section provides a detailed analysis, offering valuable insights into addressing the core challenge of utility-preserving. This convincingly motivates the necessity of the proposed EUPMU framework.\n2. This paper proposes an interesting and intuitive methodological framework. Starting from the constrained optimization formulation in Section 3, the derivation seamlessly transitions to the design of Explicit Unilateral Gradient Surgery, making its geometric interpretation highly accessible. The approximation technique, based on solving a dual problem, cleverly avoids the need for separate gradient computations for multiple objectives.\n3. The authors provide rigorous theoretical analysis for proposed method, such as the proofs for both Pareto optimality (convex case) and Pareto stationarity (non-convex case). Remark 3.6 further deduces the algorithm’s properties from these results, demonstrating that it achieves sufficient optimization of the forgetting objective while maintaining predetermined utility loss bounds. This theoretical guarantee directly resolves the core problem identified in the study.\n4. The comprehensive experimental results robustly support the theoretical claims. The paper provides extensive tests on Pareto optimality, concept removal efficacy, and computational efficiency, offering empirical evidence for the algorithm’s superiority across all key metrics.\n\n**Weaknesses:**\n1. The introduction of two new hyperparameters in EUPMU increases the complexity of practical tuning.\n2. The proposed Explicit Unilateral Gradient Surgery bears resemblance to prior techniques, which diminishes its perceived originality compared to novel contributions in the field.\n\nQUESTIONS:\n1. While the paper claims computational efficiency comparable to the linearization method, Table 5 shows that EUPMU still has a slightly higher RTE than Linearization. What causes this discrepancy?\n2. The proposed method appears broadly applicable to all machine unlearning scenarios, yet the experiments only demonstrate results on image classification and generation tasks. What is its applicability in language models?\n3. If error tolerance is disabled, would the results be comparable to those of the multi-objective optimization method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}