Title: A Reliable Cryptographic Framework for Empirical Machine Unlearning Evaluation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Unlearning

Differential Privacy

Membership Inference Attacks

Cryptography
Abstract: Machine unlearning updates machine learning models to remove information from specific training data samples, complying with data protection regulations that allow individuals to request the removal of their personal data. Despite the recent development of numerous unlearning algorithms, reliable evaluation of these algorithms remains an open research question. In this work, we focus on membership inference attack (MIA) based evaluation, one of the most common approaches for evaluating unlearning algorithms, and address various pitfalls of existing evaluation metrics that lack theoretical understanding and reliability. Specifically, by modeling the proposed evaluation process as a cryptographic game between unlearning algorithms and MIA adversaries, the naturally-induced evaluation metric measures the data removal efficacy of unlearning algorithms and enjoys provable guarantees that existing evaluation metrics fail to satisfy. Furthermore, we propose a practical and efficient approximation of the induced evaluation metric and demonstrate its effectiveness through both theoretical analysis and empirical experiments. Overall, this work presents a novel and reliable approach to empirically evaluating unlearning algorithms, paving the way for the development of more effective unlearning techniques.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a cryptographic game framework based on membership inference attack to evaluate machine unlearning algorithms. It aims to address the unreliability of existing MIA-based evaluation metrics. And the paper proposes a practical tool, the SWAP test, to approximate the proposed evaluation metric. The empirical findings reveal the effectiveness the metric.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Interesting Application of Cryptographic Games: The paper frames the evaluation of unlearning algorithms within the context of cryptographic games, borrowing the concept of \"advantage\" from cryptography. This approach aims to establish a more reliable and theoretically grounded evaluation metric, which is an interesting knowledge transfer.\n2. Practical Feasibility with SWAP Test: Recognizing the computational challenges of exact calculation, the introduction of the SWAP test provides a practical approximation method. \n3. Clear Presentation: The paper is logically structured, and the concepts are presented clearly. Weaknesses:\n1. Domain Transfer: While the application of cryptographic games to unlearning evaluation is interesting, the core methodological contribution appears to be primarily a transfer and adaptation of existing cryptographic concepts rather than the development of fundamentally new algorithmic techniques for unlearning itself. This could be perceived as more of a modeling contribution than a deep algorithmic innovation for the unlearning process\n2. Reliance on \"Weak Adversaries\": The paper acknowledges that current state-of-the-art MIA adversaries are \"weak\". If more sophisticated MIAs emerge that can leverage the relationship of data points, the effectiveness of the current evaluation framework in truly reflecting privacy risks might be compromised. The paper states that if weak adversaries improve, their metric would also benefit, but this doesn't alleviate the current limitation.\n3. Difficulty in Distinguishing Unlearning Performance under Low Noise/High Privacy Budgets: The paper's experimental results (Table 1) show that when differential privacy is applied (even with a relatively large ϵ, representing a weaker privacy guarantee), the Unlearning Quality for different unlearning methods, including \"NONE\" (no unlearning) and \"REMAIN\" (the gold-standard), can be high and quite similar. This implies that the metric might struggle to discern fine-grained differences in unlearning quality with low noise or stringent privacy budgets. However, as the ϵ parameter increases (indicating less stringent privacy), the performance differences among the various unlearning algorithms become more pronounced.\n\nQUESTIONS:\n1. Impact of Varying Sensitivity Distribution ($P_D$): The paper defaults to a uniform sensitivity distribution $P_D=U(D)$. What are the theoretical and empirical implications if $P_D$ is non-uniform, reflecting varying data sensitivities or attack focuses? How would this impact the calculated \"advantage\" and the practical utility of the Unlearning Quality metric?\n2. Distinguishing Unlearning Performance under Low Noise/ High Privacy Budgets: As mentioned in weakness 3, in Table 1, for low noise/high privacy budgets, even when the performance differences among the various unlearning algorithms become more pronounced, the Unlearning Quality for various unlearning methods is consistently high, and the differences between different algorithms appear negligible. How then can one compare the advantages and disadvantages of different algorithms?\n3. Cross-Dataset/Model Performance Variations and Metric Design: In Appendix C.4, the paper attributes significant performance variations across different datasets and model architectures to models’ learning capabilities. Do other existing MIA-based evaluation methods (e.g., MIA AUC, IC Test) exhibit similar magnitudes of performance differences across these diverse settings, or is this variability particularly pronounced with the proposed metric? If the latter, could this variability be influenced by specific aspects of the evaluation framework itself?\n4. Theoretical Bounds for SWAP Test Approximation Error: The paper introduces the SWAP test as a practical approximation method. Are there any theoretical bounds or guarantees on the approximation error of the SWAP test relative to the exact \"advantage\" metric? Could the authors provide more quantitative analysis regarding the precision of this approximation? \n5. Robustness to Advanced MIA Adversaries: If MIA attacks become more powerful, for example, by considering relationships between data and using data covariance for attacks, how could the framework be extended or adapted to incorporate such advanced attack capabilities to ensure its long-term relevance and reliability?\n\nLIMITATIONS:\n1. Assumption of Uniform Data Distribution: The framework relies on a uniform data splitting strategy for the forget and test sets. Deviations from this i.i.d. setup could complicate theoretical guarantees and empirical analysis.\n2. Reliance on Weak Adversaries: The current empirical evaluation is predicated on state-of-the-art MIA adversaries being \"weak\" and making independent decisions per data point, not exploiting data covariance. This might limit the framework's ability to reflect true privacy risks against more sophisticated future attacks.\n3. Limited Discernment in High Privacy/Low Noise Regimes: The metric struggles to show significant differentiation between unlearning algorithms when differential privacy is applied, even with larger ϵ values, as all methods achieve very high and similar Unlearning Quality scores\n4. Lack of Quantified Approximation Error for SWAP Test: While the SWAP test is proposed as a practical approximation, the paper does not provide theoretical bounds or quantitative analysis of its approximation error for the general \"advantage\" metric.\n5. Variability in Metric Performance Across Diverse Settings: The evaluation metric shows significant variability in performance across different datasets and model architectures. While partly attributed to inherent task difficulty, a deeper discussion on whether this variability is amplified by the evaluation framework itself compared to other metrics is needed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper proposes a new metric to evaluate the effectiveness of machine unlearning. Machine unlearning is intended to remove the trace of sensitive training data from a model. Existing metrics for unlearning sometimes have counter-intuitive outcome, e.g., assigning low score to models retrained from scratch without the sensitive data. This paper provides a theoretical framework as well as computationally tractable heuristics that instantiate the framework. Empirical evidence shows that the new score is consistent with the expected outcome of DP-guaranteed models across different privacy parameters.

STRENGTHS AND WEAKNESSES:
This paper is very well written. The problem setting is clearly motivated. The theoretical framework is concisely and formally defined. The heuristics are sound. The experiment designs are also smart: the authors use different levels of DP to overcome the lack of ground truth effectiveness of unlearning. Overall, I believe this paper is of high quality. 

There is no apparent weakness in the work. I do have questions about the expected behavior of unlearning score in some corner cases. I'd appreciate if the authors could clear my doubt.

QUESTIONS:
As the authors mention in the work, one shortcoming of MIA based evaluation is that score may be low for model retrained without the data to be removed. However, is this behavior expected if there are duplicates in the data set? For example, two individuals may have identical data inputs. Will MIA mistake the remained duplicate as the entry to be removed? How does new framework overcome this problem?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel, theoretically grounded framework for evaluating machine unlearning algorithms using a game-theoretic approach inspired by cryptographic games. The authors introduce the Unlearning Sample Inference Game, which models the interaction between an unlearning algorithm and a membership inference attack (MIA) adversary. They define a new metric, Unlearning Quality (Q), based on the adversary’s advantage in distinguishing forgotten from unseen data points. The metric satisfies desirable properties such as zero advantage for retraining (i.e., perfect unlearning) and alignment with certified removal guarantees.\n\nTo make the framework practical, the authors develop an efficient approximation called the SWAP test, which preserves theoretical guarantees while reducing computational cost. Empirical results on CIFAR-10 demonstrate that the proposed metric is more robust and consistent than existing attack-based evaluation metrics, such as MIA AUC and inter-class confusion, especially under varying differential privacy budgets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper introduces the Unlearning Sample Inference Game, inspired by cryptographic games, to model the interaction between an unlearning algorithm and an MIA (Membership Inference Attack) adversary. The paper proposes the concept of \"advantage\" to quantify privacy risk and unlearning efficacy, providing theoretical grounding.\n\n2. The theoretical guarantees provided are strong.\nThe framework satisfies desirable properties:\n\n- Zero Grounding: The retraining baseline achieves 0 adversarial advantage (i.e., Q = 1).\n\n- Certified Removal Guarantee: Strong connection between the metric and differential privacy–inspired certified removal.\n\n- Handles multiple MIA adversaries, addressing inconsistency in existing MIA-based evaluations.\n\n\n3. The SWAP test offers an efficient way to approximate the proposed metric while maintaining symmetry and grounding properties. Empirically shown to be more stable than naive split-based MIA evaluations.\n\n4. The paper has comprehensive experiments. The authors evaluate popular unlearning methods (e.g., Fisher Forgetting, SSD, SALUN) on CIFAR-10 with ResNet. They also benchmark performance across varying DP budgets, showing the proposed metric Q aligns well with theoretical expectations.\n\n5. Robustness and consistency are a strength as they compared existing metrics (MIA AUC and Interclass Confusion), \nThe proposed Unlearning Quality metric maintains monotonic behavior with increasing privacy budget. It also offers a more consistent ranking of unlearning methods across different conditions.\n\n6. The authors also do a thorough review of prior work in attack-based evaluation (e.g., MIA metrics) and theory-based and retraining-based metrics. They effectively position this work as addressing the pitfalls of all three categories using a theoretically justified, attack-based approach.\n\n**Weaknesses / Limitations:**\n1. Assumption of Weak Adversaries: Practical evaluation is restricted to weak MIAs (interact only once with the oracle), limiting full assessment of sophisticated attacks.\nWhile justified by current MIA limitations, this could underestimate real-world risks.\n\n2. No Real-World Dataset in that experiments are confined to CIFAR-10, a toy dataset. Lack of tests on large-scale or sensitive datasets (e.g., medical records, text corpora) limits external validity.\n\n3. There seems to be some reliance on certified removal for bounds. Certified removal parameters (ε, δ) are difficult to compute or guarantee in practice, especially for non-convex or large models.\n\n4. Although the SWAP test reduces cost, full computation of the proposed metric can still be computationally expensive in real-world pipelines.\n\n5. While the authors critique other MIA-based methods for giving false confidence, the possibility that their own metric may also be over-optimistic under strong attacks is not discussed in depth.\n\nQUESTIONS:\nHave the authors tested the metric’s robustness against white-box MIAs, like gradient-based or loss trajectory attacks? Can the proposed framework incorporate more recent adaptive MIAs (e.g., Carlini’s attacks)?\n\nDo you believe the metric would be stable across different architectures (e.g., transformers, graph models)? Could optimization landscapes affect the adversary’s success, and hence the unlearning score? Another thing is that do you think results would generalize to imbalanced datasets or different modalities? While a more empirical question, would love to hear the authors' thoughts on this.\n\n\nHow sensitive is the unlearning quality metric to non-uniform sensitivity distributions? What happens when the forget and test sets differ in class distribution or sample difficulty?\n\nLIMITATIONS:\nThe evaluation framework assumes that the adversary only queries the oracle once, i.e., a “weak adversary,” yet most real-world attackers may use multiple adaptive queries or exploit correlations.\n\nThe SWAP test depends on careful dataset splits with equal-sized forget/test sets and uniform sensitivity distribution.\n\nOnly black-box or weak MIAs are used, no white-box or shadow-model-free MIAs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a notion of unlearning evaluation defined via a distinguishing game dubbed the “unlearning sample inference game.” Informally, this game asks an adversary to distinguish unlearned data from data never used in training at all, given access to the model. The distinguishing game is presented in the style of cryptographic security games, where the adversary is assumed to be computationally bounded and the quality of unlearning is described as a function of the adversary’s distinguishing advantage. Notably, the “unlearning” method of retraining a model on a dataset from which all unlearned data has been removed guarantees that any adversary will have no advantage at all in the distinguishing game. A test for evaluating the advantage of the adversary without computing it precisely is proposed and empirically evaluated on the CIFAR10 dataset and ResNet model output by LR. Retraining, Fisher forgetting, final layer fine-tuning, final layer retraining, negative gradient descent, saliency unlearning, selective synaptic dampening, and no unlearning are evaluated.\n\nSTRENGTHS AND WEAKNESSES:\nThe distinguishing framework for unlearning evaluation is very natural and I believe captures a very reasonable notion of unlearning. I have some questions about the usefulness of this definition compared to other potential definitions, however. It would be nice to better understand how this definition compares to one where the adversary is computationally unbounded. There have been challenges in separating computational differential privacy from standard (statistical) differential privacy, for instance, and so I’m not sure I appreciate why restricting to computationally bounded adversaries is the right thing to do here, and how this changes the threat model. Some discussion of this restriction would help with interpreting the definition. \n\nThere is significant overlap between this paper and another recent paper [1] up on arxiv, so I’d recommend citing this work as well. I believe this went up after the NeurIPS abstract deadline, however, so I don’t think there’s an obligation to have compared to the work already.\n\n[1] Mirror Mirror on the Wall, Have I Forgotten it All? A New Framework for Evaluating Machine Unlearning. Brennon Brimhall, Philip Mathew, Neil Fendley, Yinzhi Cao, Matthew Green\n\nQUESTIONS:\nSee comments on strength and weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}