Title: Efficient Verified Unlearning For Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Unlearning

Verified Unlearning

Knowledge Distillation

Data Privacy
Abstract: Growing data privacy demands, driven by regulations like GDPR and CCPA, require machine unlearning methods capable of swiftly removing the influence of specific training points. Although verified approaches like SISA, using data slicing and checkpointing, achieve efficient unlearning for single models by reverting to intermediate states, these methods struggle in teacher-student knowledge distillation settings. Unlearning in the teacher typically forces costly, complete student retraining due to pervasive information propagation during distillation. Our primary contribution is PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), a novel framework integrating verified unlearning with distillation. We introduce constituent mapping and an incremental multi-teacher strategy that partitions the distillation process, confines each teacher constituent’s impact to distinct student data subsets, and crucially maintains data isolation. The PURGE framework substantially reduces retraining overhead—requiring only partial student updates—when teacher-side unlearning occurs. We provide both theoretical analysis, quantifying significant speed-ups in the unlearning process, and empirical validation on multiple datasets, demonstrating that PURGE achieves these efficiency gains while maintaining student accuracy comparable to standard baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), an unlearning framework tailored for knowledge distillation. The method proposes a student-teacher constituent mapping and an incremental multi-teacher distillation strategy to ensure data isolation during the distillation process. The approach extends SISA to allow partial student retraining when teacher-side unlearning occurs, avoiding the inefficiencies of full student retraining. Experiments demonstrate that PURGE achieves significant speed-ups in unlearning while maintaining performance.\n\nSTRENGTHS AND WEAKNESSES:\nQuality：\nThe method is sound and builds on SISA, but it inherits SISA’s scalability issues, maintaining many models and checkpoints is costly. Experiments are limited to small datasets, with no substantial evaluation on large models like BERT.\n\n\nClarity: The writing is clear and well-organized. Key ideas like constituent mapping and incremental distillation are explained logically and are easy to follow.\n\nSignificance:\nThe paper defines the student-side retraining as part of machine unlearning, but this is conceptually questionable. Machine unlearning aims to remove training data influence from a trained model, whereas the student model here never directly sees the data being unlearned. The proposed method is better viewed as a pipeline update mechanism, NOT unlearning.\n\nOriginality:\nAlthough the paper explores a new angle in distillation, it does not introduce a fundamentally new unlearning method, as it is entirely based on SISA. Since the student model does not directly unlearn any data, the work falls outside the core definition of machine unlearning.\n\nQUESTIONS:\n1. Machine unlearning refers to removing the influence of specific training data from a trained model. In this work, the student-side unlearning operations are more accurately characterized as \"cascading synchronization/update after unlearning\" rather than unlearning. This distinction is conceptually important. Could the authors clarify how they define unlearning and what distinguishes the proposed approach from standard SISA beyond the cascading synchronization mechanism?\n\n2. The term \"verified unlearning\" is used repeatedly throughout the paper. Could the authors provide a precise definition of what constitutes \"verified unlearning\" in this context, along with the formal criteria or standardized references that support this terminology? Additionally, what theoretical proofs or empirical guarantees demonstrate that the resulting model is equivalent to one trained without the deleted data, thereby justifying the \"verified\" claim?\n\n\n3. The paper assumes SISA as the default verified unlearning framework without questioning its scalability limitations, such as high training cost from maintaining multiple models and performance degradation due to data fragmentation. Is the sharding architecture truly necessary for machine unlearning, or could alternative approaches achieve similar goals without these inherent overhead costs?\n\n4. Have the authors considered the practical utility of approximate unlearning methods, and could they provide a comparative analysis of the trade-offs between exact and approximate approaches in terms of performance, efficiency, and regulatory compliance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes PURGE, an extension of the data-isolation verified-unlearning framework SISA to knowledge distillation (KD). SISA’s core idea is to shard and slice data so only affected slices retrain when deletion requests arrive. In naive KD, however, shard-isolated teachers become interdependent when soft labels are computed using all the teacher models, re-coupling the shards and potentially leaking deleted information.\n\nPURGE restores isolation by applying sharding and slicing to both student data and teacher ensemble. Specifically, it introduces a chunk-and-slice schedule: within each shard, teachers are grouped into overlapping sets, tied to disjoint chunks; the student is distilled incrementally, starting with soft labels from a single-teacher + chunk and progressively moving through additional data-chunks with more and more teachers being used in additional chunks for soft-label generation, till all the teachers are used. Each chunk of student data are further sliced, enabling checkpoint-based rollback exactly like SISA. When a teacher or data point must be forgotten, only the student slices whose training relied on chunks containing that teacher --- or on the deleted data --- require regeneration, reducing unlearning cost while preserving KD accuracy. Experiments show significant wall-clock speedups without accuracy loss.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The authors identify an important problem of unlearning in the KD setup, and the extension of SISA is natural and well explained. The clever bit is mapping teacher slices to student chunks so only a slice-of-a-student retrains when a teacher unlearns; Nice, even though incremental.\n- Experiments span a good scale of models from small models on simple MNIST datsets to models with a few 100 million parameters at BERT scale.\n- The paper is well written and conveys its ideas cleanly.\n\n**Weaknesses**\n\n- KD-per-shard baseline under-explored. A straightforward alternative is one teacher–student pair per shard. Even if inference latency rises, memory traffic, activation size, and communication overhead could make this baseline competitive on larger datasets (e.g., ImageNet-1k).\n\n- Chunk-free ablation. The paper would benefit from deeper analysis into the chunk-free ablation. Set $c = 1$ and increase the number of shards/teachers until total teachers = M. Report (a) unlearning latency and (b) accuracy. This isolates the benefit of chunking itself. Consider the chunk-free baseline where,  `#shards` $\\leftarrow$ `# chunks * # shards`. Here, the number of student models increase by `#chunks`, but no ordered training is required at the chunk level and each student works with smaller amount of data. Would this not make unlearning more efficient and the process more resource efficient?\n\n- Does the method admit any easy extension to open-domain tasks?\n\n- Stability claim: Incremental multi-teacher \"provides a more stable training process\" will benefit from substantiation beyond accuracy curves. For instance, including loss/accuracy traces or gradient-norm stats across multiple training runs/models with other aspects kept constant would be beneficial.\n\nQUESTIONS:\n1. Could you report an ablation where $c = 1$ (no chunking) but the number of teacher constituents is increased so that each student still maps to exactly one teacher?\nThis would separate the benefit of data-isolation itself from the chunking-over-time schedule.\n\n2. You claim incremental multi-teacher distillation yields “a more stable training process.” Could you provide convergence plots (loss/accuracy vs steps) or variance metrics to substantiate this?\n\n3. Does constituent mapping still give verified guarantees when the output space is unbounded (e.g., sequence generation)? If so, how would you adapt PURGE to autoregressive decoding?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a significant and timely problem (efficient verified unlearning in KD, specifically for teacher updates) with a novel and well-motivated framework (PURGE). The core idea (constituent mapping + incremental multi-teacher distillation) is sound and effectively tackles the critical information propagation issue. Theoretical analysis and thorough experiments demonstrate clear efficiency gains while maintaining accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Tackling verified unlearning specifically for teacher-side updates in KD is a crucial and previously under-addressed challenge.\n2.  The central idea of constituent mapping and the incremental multi-teacher distillation strategy is elegant, well-motivated, and effectively solves the information leakage problem inherent in naive SISA application to KD.\n3.  Experiments are comprehensive, covering multiple datasets (image, text) and tasks. The core claims are well-supported.\n4. The solution makes verified unlearning practically viable for teacher-student pipelines, which is essential for deploying and maintaining large models under privacy regulations.\n\nWeaknesses\n- The theoretical speedup Nx is a minimum under idealized assumptions (uniform sharding, uniform slices/chunks, linear scaling). Real-world overheads (checkpointing I/O, varying slice sizes, non-linear training dynamics) are ignored. The ceiling function impact is noted but its practical significance for large rc isn't deeply explored.\n\n- Soft Label Regeneration Cost: The efficiency analysis (Fig 2, Eq 1-4) explicitly excludes the time to regenerate soft labels (Y_{k,i} for i >= l) after a teacher update. This cost, especially for large models or large chunks, could be substantial and potentially dominant over the partial student retraining time in PURGE. While the paper argues inference cost is \"negligible,\" this is highly dataset/model-dependent and needs quantification, particularly for the large models mentioned. Comparing PURGE's total time (teacher update + soft label regen + student partial retrain) vs. naive SISA's total time (teacher update + student full retrain) is crucial for a fair efficiency assessment. This is a significant weakness in the current efficiency claim.\n\n- While the focus is on verified unlearning, comparing PURGE's efficiency/accuracy trade-off against state-of-the-art approximate KD unlearning methods (even if only student-side focused or lacking guarantees) would provide valuable context. How much efficiency is sacrificed for the verification guarantee?\n\n- PURGE requires storing intermediate student checkpoints per student constituent. For large models and many slices/chunks, this storage overhead could be prohibitive. This is not discussed.\n\nQUESTIONS:\n1. The analysis shows speedup decreases as c (chunks/student) increases for fixed M. The paper suggests using N=M (c=1) for max speedup. However, increasing N linearly increases the total number of student constituents to train and store, plus the overhead of aggregating N predictions during inference. The trade-off between speedup and total resource consumption (compute, memory, storage) needs explicit discussion. Is N=M always practical?\n\n2. The description of the incremental training process (Section 3.1, steps 1-3, and the state transitions S_{k,l,j}) is dense and slightly hard to follow on first read. A small pseudocode snippet could significantly improve clarity.\n\n3. The trade-off regarding r (slices per chunk) is mentioned (Sec 3.2 end, Sec 4.1) but its impact on student-side unlearning efficiency isn't quantified. How much faster is student unlearning with larger r? This is important context for choosing r.\n\n4. The term \"constituent\" is used heavily. While defined, consistently using \"model constituent\" or \"submodel\" initially might help readability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PURGE, a framework for efficient and verified machine unlearning in the context of knowledge distillation (KD). While existing methods like SISA provide provable unlearning in single model settings, They don't work well when the teacher changes, because the updates spread through the whole model and require retraining all the students.\n\nTo address above issues, PURGE introduce constituent mapping to isolate teacher influence to specific student constituents and also apply incremental multi-teacher distillation, where student constituents are trained in stages using subsets of their assigned teacher ensemble, preserving data isolation.\n\nThe authors provide both theoretical analysis (showing ≥N× speedup for N student constituents) and empirical validation across several datasets (MNIST, SVHN, CIFAR-100, SST5), showing that PURGE substantially improves unlearning efficiency without degrading predictive performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper introduces a novel problem setup by identifying a specific failure case for SISA when applied in the context of knowledge distillation.\n\n- The proposed method is clearly explained, and the logic is easy to follow, making it accessible for readers.\n\nWeaknesses:\n\n- While verified unlearning in image and text classification is important, the scope of the paper feels somewhat narrow. It only considers relatively small models like ResNet-50 and BERT. This makes the claim about transferring knowledge from \"large, computationally intensive teacher models\" feel less convincing. It would be stronger to include experiments involving large reasoning models being distilled into smaller LLMs with reasoning abilities (Just one example I can image). \n\n- The data mapping process might introduce bias depending on the models used. How is the mapping strategy chosen? Are there any optimal ways to split data that could help improve or degrade performance? A deeper discussion would be helpful.\n\n- The analysis experiments are limited. While the method does seem to address the specific failure of SISA under knowledge distillation, the paper would benefit from additional experiments designed to provide more insights into the strengths and limitations of the proposed approach.\n\nQUESTIONS:\nPlease see the questions below, which are related to the weaknesses mentioned:\n\n1. Are there any more realistic scenarios that could be considered? For example, the original SISA paper explores transfer learning from a pre-trained ImageNet model. Would a similar setting help validate the method in a more practical context?\n\n2. Could additional experiments be designed to provide deeper insights into the strengths and limitations of the proposed method?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}