Efficient Verified Unlearning For Distillation
Authors: Yijun Quan
Zushu Li
Giovanni Montana
Venue: NeurIPS 2025 poster ()
Keywords: Machine Unlearning
Verified Unlearning
Knowledge Distillation
Data Privacy
Primary area: general_machine_learning
TL;DR: A verified unlearning method provides an efficient mechanism to update models when either the student's or the teacher's data receive unlearning requests.

Abstract: Growing data privacy demands, driven by regulations like GDPR and CCPA, require machine unlearning methods capable of swiftly removing the influence of specific training points. Although verified approaches like SISA, using data slicing and checkpointing, achieve efficient unlearning for single models by reverting to intermediate states, these methods struggle in teacher-student knowledge distillation settings. Unlearning in the teacher typically forces costly, complete student retraining due to pervasive information propagation during distillation. Our primary contribution is PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), a novel framework integrating verified unlearning with distillation. We introduce constituent mapping and an incremental multi-teacher strategy that partitions the distillation process, confines each teacher constituent’s impact to distinct student data subsets, and crucially maintains data isolation. The PURGE framework substantially reduces retraining overhead—requiring only partial student updates—when teacher-side unlearning occurs. We provide both theoretical analysis, quantifying significant speed-ups in the unlearning process, and empirical validation on multiple datasets, demonstrating that PURGE achieves these efficiency gains while maintaining student accuracy comparable to standard baselines.

Efficient Verified Machine Unlearning for Distillation
Yijun Quan, Zushu Li, and Giovanni Montana
Warwick Manufacturing Group
University of Warwick
CV4 7AL. {yijun. quan, 2.1i.19, g montana, @warwick ac .k
Abstract
Growing data privacy demands, driven by regulations like GDPR and CCPA, quire machine unlearning methods capable of swiftly removing the influence specific training points. Although verified approaches like SISA, using data slicing and checkpointing, achieve efficient unlearning for single models by reverting to intermediate states, these methods struggle in teacher-student knowledge distilla- tion settings. Unlearning in the teacher typically forces costly, complete student retraining due pervasive information propagation during distillation. Our pri- mary contribution is PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), novel framework integrating verified unlearning with distilla- tion. We introduce constituent mapping and an incremental multi-teacher strategy that partitions the distillation process, confines each teacher constituent model's impact to distinct student data subsets. and crucially maintains data isolation. The PURGE framework substantially reduces retraining overhead-ree -requring only partial student updates -when teacher-side unlearning occurs. We provide both theoretical analysis, quantifying significant speed-ups in the unlearning process, and empirical validation on multiple datasets, demonstrating that PURGE achieves these efficiency gains while maintaining student accuracy comparable to standard baselines.
Introduction
Growing data privacy demands, driven by a global wave data privacy regulations exemplified by the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA), grant users the right to retract their data from machine learning models. Fulfilling these rights critical, as models can potentially memorize training data [10] 4), necessitating alterations to trained models upon data retraction requests. However, naively retraining modern deep neural networks, which can contain billions of parameters, from scratch after each data removal is computationally prohibitive and economically unviable, especially given the potential frequency of such requests. This necessitates efficient machine unlearning techniques. Furthermore, many applications demand verified unlearning methods that formally guarantee the complete removal of data influence, a property often lacking in approximate or model-specific approaches which may only offer heuristic probabilistic removal
guarantees 25 32 [9] The challenge lies in developing methods that are both computationally efficient and provably effective in removing data influence.
The Sharded, Isolated, Sliced, and Aggregated (SISA) framework [ offers a prominent solution for achieving verified, model-agnostic unlearning. SISA partitions the training data into isolated shards and trains an ensemble of constituent models, where each model learns exclusively from its assigned shard. Training proceeds incrementally data 'slices" within each shard, with intermediate model checkpoints saved after processing each slice. This inherent isolation ensures that unlearning data point typically requires only reverting the single affected constituent model to relevant prior state 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and partially retraining it on small fraction of data. This mechanism provides exact unlearning guarantees (matching the model distribution as if trained without the removed data) while offering significant efficiency gains over full retraining in standard single-model scenarios.
While SISA is powerful for individual models, adapting verified unlearning effectively more complex learning paradigms like Knowledge Distillation (KD) poses unique, previously unaddressed challenges. KD crucial in modern machine learning, enabling the deployment of sfate-o-tthe-ae capabilities by transferring knowledge from large, computationally intensive "teacher' models (often trained on vast datasets) to smaller, more efficient 'student' models suitable for resource- coource-conttrained environments or low-latency applications [15] 3 26. During this distillation process, however, information about the teacher's training data can leak and propagate pervasively throughout the student network (24]. Consequently, even if SISA's partitioning and checkpointing are applied independently to both teacher and student ensembles, unlearning data from the teacher side forces costly, complete retraining of the entire student network. This occurs because all student constituent models are exposed to influence derived from the original, complete teacher ensemble during initial training, fundamentally breaking the data isolation necessary for efficient unlearning when the teacher model is updated. This critical issue negates SISA's efficiency benefits within the coupled teacher-student system, hindering the practical application of verified unlearning in common KD pipelines.
To address this critical gap, we propose PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), a novel framework specifically designed for efficient and verified unlearning within the KD paradigm, focusing particularly on the challenge of teacher-side updates. The PURGE framework integrates SISA with KD by introducing constituent mapping, where each teacher constituent model's influence is restricted to a dedicated subset of student constituent models, and an incremental multi- teacher strategy for managing the distillation flow within student shards. This structure crucially maintains data isolation during the distillation phase itself, preventing the cross-model information propagation that plagues naive SISA applications in KD. PURGE enables efficient student unlearning: only small, targeted fraction of the student network needs retraining when teacher data is removed, thereby restoring the efficiency promise of SISA for the entire system. Evaluated on both image classification and sentiment analysis tasks using public datasets, including MNIST(8), SVH) 1EFAR-100(20 and SST5(28), our proposed PURGE delivers significant speed-ups over SISA while preserving model performance across various conditions.
Our key contributions are: (1) the first framework, to our knowledge, providing verified unlearning specifically tailored for distillation scenarios involving teacher updates; (2) the novel mapping and incremental multi-teacher mechanism designed to preserve data isolation during distillation; (3) theoretical analysis quantifying the significant retraining speedups achieved by our method; and (4) empirical validation on multiple datasets demonstrating substantial practical efficiency gains without sacrificing student predictive performance compared to relevant baselines.
2
Related Work
Machine unlearning aims to efficiently remove the influence of specific data points from trained models, driven largely by data privacy regulations and the need to correct data errors. Broadly, approaches can be categorized into (1) approximate unlearning and (2) exact (or verified) unlearning.
Approximate unlearning often rely on heuristics such as proposing unlearning as learning with negative updates 3, using influence functions or Newton updates [K] model scrubbing [1I, or leveraging connections to differential privacy (27] to estimate and counteract the contribution of data points to be removed. While potentially faster or applicable in specific settings (e.g.. convex models 13 277 ) these methods typically lack formal guarantees regarding the complete removal of data influence for general deep learning models, and therefore may fall short of meeting the strict requirements of certain data privacy regulations.
Verified unlearning methods, in contrast, aim to produce model state identical in distribution to one trained without the removed data. The Sharded, Isolated, Sliced, and Aggregated (SISA) framework 1 is a leading approach in this category. As outlined in Section] I SISA achieves verified, model- agnostic unlearning through data partitioning (sharding), incremental training on data slices, and checkpointing, enabling efficient retraining of only isolated constituent models when data is removed.

---

While powerful, the architectural assumptions of SISA mean its effective application often requires adaptation for specific machine learning paradigms beyond standard supervised learning on indepen- dent data. For instance, significant research has explored adapting SISA for Federated Learning (FI.), addressing challenges related to decentralized data and communication constraints [29]. Similarly, applying SISA to Graph Neural Networks (GNNs) necessitates handling graph structure dependencies [6]. and adaptations exist for non-differentablee models like random forests where partitioning applies to the model structure itself 22. Further demonstrating this need, Kumar et al. adapted SISA princi- ples for large NLP models by retraining only lightweight adapter layers within shards to manage the high computational and memory costs [21) This pattern highlights that achieving efficient verified unlearning often demands tailored solutions that respect the constraints and information flow of the target learning paradigm.
Knowledge Distillation (KD) presents another such paradigm with unique challenges for unlearning. As discussed, KD transfers knowledge from teacher to a student model, a process crucial for model compression and deployment. However, this knowledge transfer intrinsically creates dependencies: information about the teacher's training data can leak to the student [24], complicating unlearning. Research addressing unlearning specifically within KD is sparse and has focused on approximate methods or student-side updates. For instance, SCRUB "disobey" the original teacher on data intended for forgetting. offering no formal unlearning guar- antees and leaving the original teacher model unchanged. RKLD [3] uses supposedly "clean" reference teacher model to guide the original model (acting as a student) via reverse KL divergence to forget specific information, again lacking formal verification and relying on the availability of suitable reference model. Other related works also utilize KD or fine-tuning primarily for approximate unlearning speedups on the student side [8] 17]. To our knowledge, no existing method provides ef- ficient, verified unlearning directly applicable to the teacher model within a KD pipeline using SISA's partitioning principles. Thus, the most direct verified approach, applying SISA independently to both ensembles, fails for teacher-side unlearning, as information propagation during initial distillation forces costly full retraining of the student network whenever the teacher model is updated.
2 trains new student model to selectively
Our work, PURGE, directly addresses this gap by proposing the first SISA-based framework, to our knowledge, specifically designed for efficient and verified unlearning in KD, particularly handling teacher-side updates. Unlike approximate methods [3] [T, (7 [22] 30 18 [7]. PURGE provides exact unlearning guarantees inherited from SISA. Critically, unlike the naive application of SISA to KD, and distinct from prior KD-uulearning techniques that focus on student retraining or require reference models, PURGE employs constituent mapping and incremental multi-teacher distillation strategy (detailed in Section 3) to maintain data isolation during the distillation process while preserv- ing student model performance. This structural modification prevents the information propagation problem and enables efficient, partial retraining of the student network when the teacher unlearns, making verified unlearning practical in sacherr-ttudnt pipelines.
3 Methodology
The effectiveness of the SISA framework comes from the data isolation introduced by making each constituent model only trained its corresponding shard without access to data in other shards. Such isolation is broken when the teacher network is used to train the student network in the standard distillation setup described previously. By using the teacher network an entirety (providing a single supervisory signal derived from the full teacher ensemble), every constituent model of the student network gains indirect access to information influenced by all the data used train the teacher network. As a result, any change in the upstream teacher network, such as unlearning data point, necessitates updates that propagate to every downstream student constituent model, mandating the full, costly retraining of the student network. Clearly, the key to addressing this problem and restoring unlearning efficiency is to maintain isolation between the influence of different teacher data shards within the student network's 's training process well. Thus, we propose a student-teacher constituent mapping strategy designed to isolate the influence of data associated with specific teacher constituent models to only a limited subset of student constituent models.
3

---

Student D
D?
Dj
1
Di
17.1
Dia
DI.
17
D
$/.
10
127
D5.1
105
103
25.
125.
DN
V
Vin
Ya
Yx
Vva
8
Aggrepation
R
Figure 1: Overview of the proposed framework (PURGE) integrating SISA with knowledge distil- lation for efficient, verified unlearning. 'hhe structure maintains data isolation during distillation, enabling efficient student retraining upon teacher updates. Key steps: (1) Sharding: Student data D partitioned into N shards (DE), each assigned
student constituent model (Sx). (2) Mapping: Each S₁ is mapped to a distinct teacher ensemble T. = {Tk.1, Tk.ca J. (3) Incremental Distillation: The student shard D is processed in Cj sequential chunks (D Crucially, soft labels (Yx,)) for chunk I are generated only by an incrementally growing teacher subensemble Fx.1 = Usefg Te.s. limiting information propagation from the full teacher ensemble. (4) SISA Slicing & Training: Each resulting data-label chunk DI_ [ Yg.1] is further divided into R₁ slices (D),,,). Sş trains incrementally these slices using standard SISA checkpointing. (5) Aggregation: Final predictions are aggregated from all trained student constituent models {Sa}. This design ensures that unlcarning affecting teacher Th.s only requires partial retraining of the corresponding student Sk.
3.1 The PURGE framework
The PURGE framework maintains the data isolation required for efficient unlearning within the distillation process by implementing strategy based on mapping specific teacher constituent models to specific student constituent models. This prevents the problematic information propagation identified in Section 2 Figure T illlustrates the PURGE framework.
We consider setup with M teacher constituent models, {Tı, T2...., Tar and student constituent models, S SN }. where M and N are not necessarily equal. Our core idea within PURGE teacher constituent models into N disjoint ensembles. Ty.... .N. such that
to partition the each student constituent model Sy learns exclusively from the teachers in its assigned ensemble Ft Let Th = (Tx,1, T .... Th.eul } be the ensemble assigned to St, containing e teacher constituent models (this number can vary per student). Crucially, each teacher constituent model Tn belongs to exactly one student's 's teacher ensemble: Oke[N] 9h and UKE[N) T, = (Ti...., TMr). This strict mapping ensures that if unlearning affects single teacher constituent model Tr only the corresponding student constituent model S₁ will potentially need retraining.
To implement the distillation under this mapping, we first follow SISA by dividing the student dataset DS into N disjoint shards {D$ (DS..... D (Uke[n) Di DS DEEIN 0), where shard D is used for training Sg. The PURGE framework then introduces one further level of partitioning
(Chunking) specific to design and applies SISA's Slicing methodology within these chunks:
1. Chunking: Each student data shard DE is further divided into e ordered, disjoint data chunks {DA. . D& kh.c. }. where e₁ is the number of teacher constituent models mapped to Sa.

---

2. Incremental Teach-re Distillation: Soft labels for training S₁ are generated chunk by chunk using progressively larger subensembles of Th. For the 1th chunk - the soft label Yg,1 is generated using only the first / teachers the assigned ensemble: YE.1- I where Fh.- - Usejt TR This incremental approach further limits the scope of influence of each individual teacher constituent model Tk, primarily to chunks 2 i.
3. Slicing: Each combined data and soft-label pair chunk, denoted Di,, [D Yx,,), is then further divided into Rg., disjoint slices "DA......,,Plu. (U)E(Ra.d) P = Di,
NJE( (iE(B...)P... = 0), analogous to standard SISA slicing.
This hierarchical structure within PURGE (shards - chunks - slices) allows for fine- -ra-ggained checkpointing and efficient unlearning. The student constituent model Sy is trained incrementally over both chunks and slices, following the SISA principle. Training starts with the first slice of the first chunk 1.), from an initial state Sy,0 The model state is checkpointed after completing training for each slice. Let Sk.I.j denote the model state after processing slice j of chunk l. To obtain Sk.ij. the preceding state (either Sk,Lj-1 if j > or 1,. if 1, > 1) is trained for et, epochs using the cumulative data processed so far, which includes all data from chunks to 1 - 1 plus slices 1 to j of chunk I: (U :) U (Ug_1Dh..o ). The loss for data-label pair [d, v) from a slice is typically standard distillation loss, (d)) y). This process continues until the final student constituent model Sx = Sk.ek.Re,r, Rr,,, is obtained after processing all chunks and slices derived from shard D.. All intermediate states Sk are stored to facilitate the fast unlearning process provided by PURGE, as detailed in Section 3.2
Finally, after all student constituent models (SI..... SN} are trained via the PURGE framework, straightforward, non- trainabiabl aggregation function (e.g., averaging the output predictions or logits) is applied during inference to produce the overall output of the student network S. To enhance clarity for the reader, we present the PURGE training procedure in pseudo-code in Appendix A A.
3.2 Unlearning process
The PURGE framework provides mechanisms for efficiently handling unlearning requests targeting either the student's training data or the teacher's training data. A detailed discussion on handling simultaneous unlearning requests for both is provided in the Appendix A.3
Unlearning student data When a request involves removing a student data point dy (and its corresponding
soft label yw) located in slice D Pi... PURGE follows the standard SISA unlearning procedure for the affected student constituent model Sy. The network state is reverted to the previously saved checkpoint Sk,,j (the state before slice D k,. was first processed). Retraining then commences incrementally from this point onwards, using the modified data slice (excluding (du, yu)) and all subsequent slices and chunks for that constituent model. This ensures that the unlearning process for student-side side data inherits the efficiency benefits of the SISA framework, requiring only partial retraining of a single constituent model.
Unlearning teacher data A key challenge addressed by PURGE is handling unlearning requests for data used train the teacher models. Suppose a data point d, is removed from the training set originally used for a teacher constituent model Tx.1 (which belongs to the ensemble T. mapped to student Sz). The unlcarning process within PURGE proceeds as follows:
1. The teacher constituent model Tk, is updated to Th, (presumably using SISA efficiently the teacher also uses it).
2. All soft labels generated by teacher subensembles that included Te. must be updated. This affects chunks I,I + 1,..., C for student Sg. Specifically, the soft label sets YL, (for (x)) need to be regenerated using the updated teacher Th, within the respective
subensembles Thi This results in updated data-label chunks D for i e [4, ck].
5

---

3. The affected student constituent model S₁ must revert its state. Since the distillation process for chunk was the first to potentially use Tx. the student reverts the state saved just before processing chunk I, which is Sk,-1 (equivalent to S1,,-1,Ru.-1>>
4. Student Sş. resumes incremental training from chunk / onwards, using the regenerated data-label chunks DV for i E [,, cu) and their constituent slices. The final updated student constituent model is denoted Sy.
This procedure ensures that the influence of the teacher's 's unlearned data d, is removed from the student network Sg, while only requiring partial retraining of that single student constituent model. The corresponding pseudo-code peeudo-coee can be found in Appendix A.I
Efficiency analysis of teacher unlearning We analyze the efficiency gain of PURGE compared to a naive SISA application for teacher-side unlearning. In the naive case, as argued in Section 2 any teacher update requires retraining the entire student network. This takes time equivalent to training the SISA student ensemble from scratch, denoted faisa- Assuming training time scales linearly with the total number of data points processed (epochs x dataset size), sisa is proportional to e' D, where D = | is the student dataset size and e' is the equivalent number of full-dataset epochs reflecting the total computational effort used for initial training.
For PURGE, only the affected constituent model Sş retrains partially. We consider an idealized scenario with even distribution: N student constituent models, M teacher constituent models, c M/N chunks per student shard, and r slices per chunk (total R = er slices per shard). We assume each slice is trained for en epochs, where en c+t e' relates the per-slice epochs to the equivalent full-training epochs e' based on total computational effort []]. When unlearning affects teacher T Sy retrains from chunk onwards. The average number of slice-processing steps required, K, averaging over which chunk 7 E [1,c] is affected, is derived in the Appendix and given by:
(((ir + 1) + (cr)((c))) i)r)
K
2
The retraining time for PURGE, fPURGE, is proportional to the number of slice-processing steps times the number of data points per slice (er)). Thus, tYURGE x K Ner The theoretical speed-up of PURGE over naive SISA is then:
fxisa tPURGE
e'D
(2)
K Ner
el and the expression for K (details in Appendix), this simplifies to: Substituting en cr+i
2
Lissa
6c2r 6c
N.
fPURGE
4c2r + 3cr + 3c r
As shown in the Appendix (Equation [0], the second factor is greater than for all positive integers and c. Therefore, PURGE provides a speed-up of at least N x compared the naive SISA approach for teacher-side unlearning. Expressing this in terms of the total number of teacher constituent models M Ne:
Iaisa
6cr + 6
= M 4c 3cr + 3c +
tPURGE
This factor decreases as e (chunks per student) increases for fixed M and r. This implies that for a fixed number of teachers M, the speed-up is maximized when is minimal (ideally e 1), which corresponds to having more student constituent models (N MM)
For unlearning request sent directly to the student side, the efficiency is determined by the total number of slices per shard, At c r, the product of chunks (c) and slices per-chunk (r). In this case, the soft-labels do not require updates and the unlearning procedure follows the standard SISA approach. As result, the expected unlearning cost for single request can be analyzed as in SISA[I]: it is proportional to 2 + 1 times the cost of retraining the full shard. Therefore, increasing r leads 3R
to a larger which in turn reduces the unlearning time for student-side requests. The speed-up approaches maximum of of 1.5x compared to a shard with no slicing. This introduces trade-off between student-side unlearning speed and the teacher-side unlearning: a larger will improves student-side unlearning speed but slows down teacher-side unlearning. Consequently, the optimal

---

Table 1: Datasets and model architectures used in the experiments.
Dataset
Type
Size
Classes
Model architecture
MNIST 8
Image 70,000
10
Conv 2 FC Layers
SVHN 12
Image
630,420
10
Conv + 2 FC Layers
CIFAR-100 20
Image 60,000
100
ResNet50 14
SST5 28
Sentence 11,855
5
Qwen2.5-7BB31 31] & BERT
choice of r depends on the expected ratio of student-side teacher-side unlearning requests, which application dependent.
The detailed derivations for Equations are presented in the Appendix.
Rationale for incremental multi-teacher distillation A core component of PURGE is the in- cremental multi-teacher training within each student shard, where the subensemble Fu.1 grows as training progresses through chunks = 1,..., Cs. This ensures teacher Tr, 's influence is primarily limited to data processed from chunk i onwards. One might consider an alternative: using only single, different teacher constituent model Tx. to generate soft labels y-aingle (D,) for each chunk l. Intuitively, this might seem to isolate influence even further.
2,
However, regarding unlearning efficiency for teacher updates, this alternative offers no advantage. If teacher Tk.I requires unlearning, the student Sg must still revert to state St and retrain from chunk onwards, regardless of whether chunk was trained using only Tr, or the subensemble Fh,1- The number of retraining steps remains the same, and since retraining time is dominated by model training rather than the (usually negligible) difference in inference time for generating soft labels (single vs. subensemble), the overall unlearning time similar for both approaches.
Crucially, however, the incremental multi-teacher approach provides more stable training process for the student constituent models. Learning sequentially from potentially diverse single teachers (Tx,1, then 22.2, etc.) can introduce abrupt changes the supervisory signal, destabilizing training. The incremental ensemble I smooths these transitions by gradually incorporating new teachers while retaining previous ones, leading to better convergence and performance, as demonstrated in our experiments (Section 4). Thus, the incremental multi-teacher strategy is adopted in PURGE.
Honest Service Provider Assumption Our proposed PURGE framework focus providing verified unlearning guarantees in knowledge distillation settings, addressing unlcarning requests that target either the teacher or the student model. When such an unlearning request occurs, the impacted constituent model is reverted to a previously saved checkpoint, before the influence of the removed data, and this data point is fully excluded from the subsequent training. This approach is both auditable and provable: assuming an honest service provider, the authority can audit the code confirm the proper execution of PURGE training and unlearning. thereby ensuring verified unlearning. While these guarantees may not hold with dishonest service provider, our proposed method is orthogonal to the methods enforcing the execution of the exact unlearning algorithms.
Experimental results
We conducted experiments on both image and sentiment classification tasks using the MNIST, SVHN, CIFAR- 100 and SST5 (detailed in Table [ to evaluate the effectiveness of PURGE. The image classification tasks were conducted on a machine equipped with one NVIDIA RTX 3090 GPU (24GB VRAM) while the sentiment classification task was conducted on machine with 8 NVIDIA A100 GPUs (80GB VRAM each) to support the large number of BERT constituent models.
4.1 Unlearning speed evaluation
We first evaluate the speed-up PURGE provides to the student network when unlearning requests target the teacher network's 's data. Following the setup described in Section 3.2 we assume each slice trained for the same number of epochs, eR- This related the equivalent full-dataset training epochs e' by en ret1 [1), ensuring comparable total computation during initial training. For this experiment, set e' 120 and evaluate on the MNIST dataset. We measure the wall-clock
7

---

retraining time required for student networks trained using PURGE and compare it against baseline representing a naive SISA application, where the student network must be fully retrained upon any teacher update. In both cases, the teacher network itself is assumed to use SISA, allowing its own updates to be efficient.
Figure 2 plots the student retraining time against the number of teacher-side unlearning requests processed. For simplicity, we refer to the baseline naive SISA approach as SISA. As our focus the efficiency gain for student retraining provided by PURGE, the times shown exclude the retraining time of the teacher network itself (which is assumed efficient via SISA in both scenarios) and the inference time required by the teacher to generate updated soft labels. While PURGE's constituent mapping allows for faster soft label regeneration compared to the baseline (where the full teacher ensemble always contributes), this inference time is typically negligible compared to retraining time and is thus excluded from our comparison.
We configured the teacher network with M = 32 constituent models. For the haseline SISA approach, where full student retraining always required upon teacher update, the student retraining time is independent of the number of student constituent models N. We thus use N = 8 constituent models for the baseline measurement. For PURGE, we varied the number of student constituent models N from 1 to 32. We tested two configurations for the number of slices per chunk: r 1 and r Note that while the theoretical analysis uses eR rc+1 e', our practical implementation uses
eR = [ the theoretical minimum, particularly for larger re values (i.e., smaller N or larger r),, potentially leading minor deviations from the theoretical speed-up analysis derived using Equations 4 We simulated 100 sequential unlearning requests targeting randomly chosen teacher constituent models. For generality, we assume the teacher and student datasets distinct (Dr Dss), so only teacher models and subsequently soft labels are updated, while the student dataset DS remains unchanged. Figure 2[a) shows the cumulative retraining time, and Figure ( shows the measured average speed-up relative to the baseline SISA.
e'T. This ceiling function means PURGE might perform slightly more computation than
Clearly, for both values, PURGE's retraining time decreases (and speed- up increases) as the number of student constituent models N increases. With
80k
40
Cumulativve Time/
60k
32
Speed
24
40k
16
N 32 (where e M/N 1), PURGE requires averaging 17 + 0.17 seconds to handle unlearning request, achieving x 32x speed-up over the baseline S/SA, which requires averaging 737.14 + 10.08 seconds for each unlearning request. The empir- ical average speed-up values in Fig-
20k
Peatund
% 20
40
60
80
100
16
24
32
Number of Unlearning Requests
Number of Students
80k
40
60k
32
Cumulative
Speedi
24
40k
16
ure ()b) closely align with the theo-
20k
retical prediction from Equation (4)
(plotted as the red curve). Minor de- viations are consistent with the use of the ceiling function for eR. being slightly more noticeable for 4 and small N (large e), as predicted.
% 20
40
60
80
100
0
16 24
32
Number of Unlearning Requests
Number of Students
Figure 2: Speed comparison of the student network update process for 100 unlearning requests sent to the teacher net- work with 32 constituent models (M = 32) on MNIST dataset. Top row: r = 1; bottom row: r = slices per chunk. Left column: cumulative processing time. Right column: measured average speed-up over naive SISA (red curve fol- lows Eq.
Comparing the results for r = 1 and r =4 4, we observe that a larger (more slices per chunk) can lead to a slightly smaller speed-up when un- learning teacher data. This stems from the incremental training structure: re- training involves recomputing later slices more often (see Eq. 1] and with large r, these later slices
incorporate more preceding data. Equation 4 confirms this dependency on r. It is important to note the trade-off: while larger might slightly slow down student retraining for teacher unlearning requests, simultaneously accelerates student retraining for student unlearning requests, because the standard SISA efficiency depends on the total number of slices (R cr). The optimal choice of
8

---

nt. 8
nt 16
nt 32
Accuracy 1%
Accuracy
Accuracy
MINIST
07
07
16 Number of Students
Number of Students
Number Students
Accuracy %
85
Accuracy
Accuracy
80
SVEN
80
-3
700
Number of Students
Number Students
Number of Students
Figure 3: Comparison of student network accuracy on MNIST (top row) and SVHN (bottom row). Accuracy is plotted against the number of student constituent models (N) for different teacher ensemble sizes (M 8, 16, 32). The plot shows results for PURGE, the SISA baseline student, the original Teacher ensemble, and the Single-teacher Soft Label ablation.
therefore depends on the expected frequency ratio of teacher versus student unlearning requests for specific application.
Overall, this experiment demonstrates that PURGE substantially accelerates student network retrain- ing when teacher-side unlearning occurs, validating our theoretical analysis. The efficiency gains scale predictably with the number of student constituent models, confirming the effectiveness of the proposed partitioning and mapping strategy.
4.2 Performance evaluation
Having demonstrated PURGE's effectiveness in accelerating retraining, now evaluate whether these efficiency gains compromise the student network's predictive performance. We compare the performance of student networks trained using proposed framework against key baselines and an ablation. The baselines are:
Teacher: The original teacher network ensemble, trained using the standard SISA pipeline. This represents an upper bound on performance expected via distillation.
SISA: A student network ensemble trained using the standard SISA pipeline, where each student constituent model learns from its data shard and soft labels generated by the aggre- gated output of the full Teacher ensemble. This represents the naive SISA application to distillation that PURGE aims to outperform in terms of unlearning efficiency.
We also compare PURGE against an ablation using single-teacher soft labels (Single-teacher), where soft labels for chunk are generated only by teacher Tk., For the mulii-teacher aspects within PURGE (and the ablation), we use simple averaging teacher outputs to generate soft labels. While more advanced multi-teacher distillation techniques exist, averaging serves as a clear baseline for evaluating the structural benefits of PURGE without confounding factors. The datasets and corresponding model architectures (Simple CNN for MNIST/SVHN, ResNet50 for CIFAR-100, BERT and Qwen2.5-7B for SST5) are detailed Table The experimental results on CIFAR-IOM and SST5 are shown in the Appendix.
Results on MNIST and SVHN We investigated performance on MNIST and SVHN datasets, varying the number of teacher constituent models M (8, 16,32) and, for each M, varying M/N chunks per
the number of student constituent models N from up to M (implying student). Figure 3 presents these results. Both teacher and student networks were trained on the full training sets with data evenly allocated across shards and chunks. The results show that PURGE achieves performance very similar to baseline SISA student, with only a minor degradation compared to the original Teacher. For instance, with M = 32 teacher constituent models and N = 32 student constituent models (c = 1), PURGE achieves 16% and 83. 33.0%% accuracy on MNIST and SVHN respectively, while the SISA baseline attains 97.08% and 83.4 33.44%. This confirms

---

that PURGE's structural modifications for unlearning efficiency do not significantly compromise predictive performance on these tasks.
As expected, accuracy tends to decline slightly for both PURGE and the SISA baseline as the number of student constituent models N increases. This is attributable the reduced amount of training data available to train each individual constituent model. However, this trend does not hold for the Single-teacher ablation (shown in Figure 3) This ablation shows substantial performance degradation compared to the Teacher and SISA baseline, particularly when N is small (i.e., when each student constituent model learns from many different single teachers sequentially across chunks, c = M/N is large). This performance drop stems from the instability induced by abrupt changes in the supervisory signal when switching between different single teachers for consecutive chunks. In contrast, PURGE's incremental
strategy smooths these transitions by gradually incorporating new teachers into the subensemble TE.1, stabilizing the training process. Quantitatively, when learning from an M = 32 teacher ensemble with only N = 1 student constituent model (c = 32), PURGE's accuracy drop relative to the SISA baseline is minimal (0. (0.14% on MNIST, 0.18% on SVHN), whereas the Single-teacher ablation suffers significant losses (1.30% MNIST, 7.35% SVHN). This validates the design choice using the incremental multi-teacher within PURGE for maintaining performance.
5
Conclusions
The need for efficient, verifiable machine unlearning is critical, especially for KD used in deploy- ing large models. However, applying verified frameworks like SISA naively to KD is inefficient for teacher-side unlearning, because information propagation forces costly full student retraining, negating SISA's benefits. To address this critical gap, we introduced PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), a novel framework integrating the principles of SISA with the specifics of KD. By employing constituent model's influence is restricted to specific student constituent models and utilizing incremental multi-teacher distillation strategy within each student shard, this framework successfully maintains data isolation throughout the student's 's training process.
Sropinggwhereee each teacher constituent
Our theoretical analysis and empirical evaluations on image and language tasks demonstrate that our method achieves its primary objective: it enables efficient, verified unlearning even when teacher data is removed, requiring only partial retraining of the affected student constituent model(s). This results in substantial speedup (at Icast N x, where N is the number of student constituent models) compared to the naive baseline. Furthermore, the approach naturally retains SISA's efficiency for handling student-side unlearning requests and, crucially, maintains student performance comparable to the standard SISA baseline, validating the stability provided by the incremental multi-teacher strategy.
By ensuring efficient and verified unlearning within reacher-studen pipelines, this capability makes the responsible deployment and maintenance of distilled models significantly more practical, particu- larly for systems involving large foundation models as teachers. Future research could explore several promising directions: integrating more sophisticated multi-teacher distillation algorithms within this structure to potentially enhance student learning efficiency and final performance; extending the theoretical analysis to cover different data distributions aggregation methods; and applying and evaluating the framework in complex, large-scale distillation scenarios involving state-of-the-art vision language models.
Acknowledgments
This research was financially supported by Engineering and Physical Sciences Research Council (EPSRC), United Kingdom, (SUSTAIN Manufacturing Hub EP/S018107//
GM acknowledges support from UKRI Al Turing Acceleration Fellowship (EPSRC EP/V024868/1),
References
[1] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Cin Chho, Hengrui Jia, Adelin Travers. Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
10

---

[2] Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings Machine Learning Research, pages 1092-1104, PMLR, 18-24 Jul 2021.
[3] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463-480, 2015.
[4] Nicholas Carlini, Chang Liu, Ulfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267-2844 2019.
[5] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.. 2017.
[6] Min Chen, Zhikun Zhang. Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC conference on computer and communications security, pages 499-513, 2022.
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: /003:/8000055533 2018.
[8] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 1411114 2012.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186 2019.
[10] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and hasic countermeasures. Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-13333 1333, 2015.
[11] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9304 9312, 2020.
[12] lan Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Amoud, and Vinay Shet. Multi-digit number recog- nition from street view imagery using deep convolutional ncural networks. arXiv preprint arXiv:1312.6000 2013.
[13] Chuan Guo, Tom Goldstein, Awni Hammun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning. pages 3842-3842, 2020.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 770-778, 2016.
[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 503.005510333311 2015.
[16] Edward Hu, yelong shen, Phillip Wallis. Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations, 2022.
[17] Hyunjune Kim, Sangyong Lee, and Simon S Woo, Layer attack unlearning: Fast and accurate machine unlearning via layer level attack and knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21241-21248, 2024.
[18] Junyaup Kim and Simon S. Woo. Efficient two-stage model retraining for machine unlearning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 4360-4368, 2022.
[19] Yongwoo Kim, Sungmin Cha, and Donghyun Kim. Are truly forgetting? critical re-exanination of machine unlcaming evaluation protocols. arXiv preprint arXiv:2503.065: 06699, 2025.
[20] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research).
11

---

[21] Vinayshekhar Bannihatti Kumar. Rashmi Gangadharaiah, and Dan Roth. Privacy adhering machine un-learning in nlp. In Findings the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 268-277, 2023.
[22] Meghdad Kurmanji. Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. In Advances in Neural Information Processing Systems, volume 36, pages 1957-1987. Curran Associates, Inc., 2023.
[23] Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1 15, 2023.
[24] Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? Advances in Neural Information Processing Systems, 36: 1037-11048, 2023.
[25] Youyang Qu, Xin Yuan, Ming Ding, Wei Ni, Thierry Rakotoarivelo, and David Smith. Learn to unlearn: A survey machine unlearning, arXiv preprint arXiv:2305.07314 2023.
[26] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[27] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithus for machine unlearning. In Advances in Neural Information Processing Systems, volume 34, pages 18075-18086 Curran Associates, Inc., 2021.
[28] Richard Socher, Alex Perelygin. Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[29] Ningxin Su and Baochun Li. Asynchronous federated unlearning. IEEE INFOCOM 2023 IEEE Conference on Computer Communications, pages 1 1-10, 2023.
[30] Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence-bar knowledge distillation for unlcarning personal information large language models. arXiv preprint arXiv:2406.00 0,998, 2024.
[31] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lui, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412. 15115, 2025.
[32] Kairan Zhao, Meghdad Kurmanji, George-Octavian Octavian Bärbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. Advances in Newral Information Processing Systems, 37:12293- -12333, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our proposed method PURGE (presented in Section 3) provides an efficient
verified unlearning method that direetly addresses potential information leakage from teacher to student knowledge distillation, as outlined in the abstract and introduction. Experimen- tal results unlearning speed evaluation and performance evaluation presented in Section support this claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We did not use a standalone section dedicated to limitations, but such discus- sions are integrated throughout the presentation. We briefly discussed the challenges of ensemble learning in multi-teacher distillation in Appendix An. suggest that a potential way improve the proposed PURGE framework is to improve the soft-label aggregation strategy while keeping the data isolation. Another key limitation the work the computa- tional overhead associated with retraining from checkpoints. Since the proposed PURGE framework is a SISA-based verified unlearning method, it inherits this limitation from SISA and similar verified unlearning approaches. While we did not explicitly emphasize this point, it is acknowledged in Section [ where we compare the verified unlearning methods against approximate unlearning methods, admitting that approximate unlearning methods could be faster but lack formal guarantees for the removal of the data. This trade-off is sell-established in the literature, and due to page constraints, did not elaborate further.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
13

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provided the theoretical proof of the speed-up that the proposed PURGE
framework provides in Supplemental Material A.2 with assumptions clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All the details for the proposed framework are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
14

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The experiments are done on 4 public datasets, and the code is released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
hubblc//gulles/CoeeSuuminsssss sose for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips t/nips.cc/publl pcc//ubllc/ggudess htta://ggiides/ooeeSSubbiiii wwlbbmisssiooioiiiiioooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details like the number of epochs are provided in the paper, or otherwise included in the publicly available code. We standard train, test, and validation splits from public datasets. The random seeds used for data splits across constituent models are included in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
15

---

Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The standard deviation of the the time required to handle each unlearning request is reported in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The specification of GPUs used for the experiment on both image and sentiment classification tasks is reported. The time of execution (unlearning time), key performance metric of the proposed PURGE framework, is clearly presented in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Code of ethics read and understood by the authors.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
16

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper proposes a verified unlearning framework for knowledge distillation, method designed to safeguard data privacy, which represents positive societal impact. The potential positive economic and environmental impact also discussed in the Appendix. To the author's ' knowledge, it does not pose any immediate negative societal impact.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
17

---

Answer: [Yes]
Justification:
The datasets used in the paper are properly cited and credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code submitted is well documented and structured.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The method developed does not involve LLMS as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to our LLM policy //tp:::////erraie cc. fc//oonferencee/2225////// for what should or should not be described.
19

---

A Supplemental material
A.1 Pseudo-code For PURGE
Algorithm 1 Algorithmic outline of the PURGE training framework
1: for k == N do
2: Divide D into C ordered, disjoint chunks (D k,1 Di.c.)
3: Initialize student model state Sk,0
4:
for 1= Cą do
5:
Generate soft labels YR, for chunk D using teachers Tk.1 = U/ This R.l
Pair chunk data and labels: D/_, = [D Ya,l
6:
Partition D into Rx, slices (D) k D D
7:
8:
for j = 1 to Rx., do
9:
Select previous state: SE.L if else SEI-1, ifl 1; else Sk.0
Gather cumulative data: ( (U,
For et, epochs, do:
12:
Compute distillation loss ((S v) for each [d. v] in cumulative data Update model parameters via gradient descent
13:
14:
Store updated intermediate state Sk, for efficient unlearning end for
15:
16:
end for
17: Final model for shard: S₁ Sk,ce.Ble.y
18: end for
Algorithm 2 A Algorithmic outline for student-side unlearning when unlearning request sent to the teacher-side in PURGE
1: Update teacher constituent model: Tk. is the current teacher constituent model
2: Te, is obtained by SISAUpdate(TT..t remove data point d,))
3: for i= to Cą do
4: Generate soft labels Yg,; for chunk using updated teachers Ts,s (including Tki))
5: Pair chunk data and labels: D [D Y2.l]
6: end for
7: Revert student constituent model: set S to checkpoint before chunk /
8: Initialize S < Sk.
9: for i = to do
10: Incrementally train S using D and slices
11: Update and store intermediate state Sk
12: end for
return S
A.2 Proof of speed-up when teacher unlearns
We follow the same setup for the epoch number as done in SISA with the same number of epochs en for each round of training on all slices. With incremental training applied on the slice level, with training progressing through the slices, the number of data points cover in epoch is increasing. We consider evenly distributed data, slices, chunks, and shards for a student network with student constituent models {S.). When the (th teacher for the 4th student is changed, the student constituent model S₁ will reverse back to Sk,1-1 and retrained from (I - 1)r + 1 chunk to ch chunk. Thus, the total number of slices the retraining process needs run is:
(((1- 1)r + + (1- 1)r)
K(I) Σ
(5)
2
where e is the number of chunks per shard and is the number of slices per chunk.
20

---

Assuming that every teacher shares the same probability of receiving an unlearning request. the average number of data points for student to retrain when an unlearning request sent one teacher can be calculated as:
K
K())
1=1
(((1 ((11- [[1 + 1) + crr - -(1 (1-111 1)r)
(6)
2
1m1
CR
(1 +cr)((c-11)
1=0
With the assumption that the training time is solely dependent on the amount of training data and each network is initially trained for equal time from scratch, we consider data set of size D trained for e' epochs. As the proposed framework should be trained for equal time, for network with N student constituent models, this gives:
D Nor
(cr + 1)D
e'D = N EER-N
ER-
Σ 1=1
ER = ER 2
where Dstice Nre is the number data points in one slice and Σ;= ER is the total number of slices a student constituent model would run through over the entire training process. It can be easily derived that
2
CT +1 d' eR
8
Given that SISA requires full retraining when an unlearning request is sent to teacher constituent model and requires training on data points, the ratio of retraining time for SISA against the proposed method can be computed as:
ÊSiSA
e'D
couple
K D
e'D
(lr+))+or)(e-Iir D
en
2
Ner
(9)
1 2
Σ
((++ ((r+1))
cort1
2
Ner
1=0
6 + 6c
N.
4c + 3er 3c +3
can be shown that the proposed method provides at least N speed-up by showing that the second part of the expression in Equation (9) is bigger than 1:
6 + 6c
2c2r 3c 3cr - 4c + 30r + 3c - r +
4c + 3cr + 3c + 1
r(2c 1)(c 1) 3(c 1) r(dc
(10)
2
:'02
For evenly distributed chunks, we have N M/c. From Equation 10 we can write the ratio for speed-up in terms of the number of teacher constituent models M and the number of chunks per student as:
tsisA
6er 6
M
(11)
eeoppee
2e + 3cr + 3c +3
For the second part, its derivative with respect to cs::
6er46
(4c2 1) Scr +
(12)
dc
(c+1))( +)3
21

---

which is negative for all positive integer and c. Thus, we will have less speed-up when we have more chunks for each student constituent model. When the number of teacher constituent models is fixed, this means that by having more student constituent models, we can have faster retraining process.
A.3 Handling simultaneous unlearning requests
In Section we discussed how the proposed PURGE can address individual unlearning requests directed at the student's or the teacher's data. In scenarios where the teacher and student potentially share the same underlying dataset (D² = DD, common in self-distillation or when using public dataset), an unlearning request might require removing a data point d, simultaneously from both networks. The PURGE framework efficiently handles this as well. We consider two main cases:
Scenario aligned data removal. Suppose the data point d, to removed exists student slice DE.,, and also corresponds exactly to data originally used to train the teacher constituent model TE.1 (i.e., the teacher responsible for generating labels starting from chunk in the affected student constituent model S₁). The unlearning process be combined:
1. Teacher Tr updates to Th.
2. Soft labels Yı.; for i E [1, ck. are regenerated using updated subensembles including Th.
3. The student constituent model Sş reverts state Sk,1-1 (due to the teacher change affecting chunk 1 onwards).
4. Sz retrains incrementally from chunk l onwards, using the updated soft labels *and* ex- cluding d, from the relevant student slice D The effective slice becomes Di..s DR..I. fidu, yu]}.
The retraining time is dominated by the steps required for the teacher update (reverting to Sa. and is nearly identical to handling only the teacher unlearning request, ignoring the negligible effect of removing one data point from the student's retraining path.
Scenario 2: misaligned data removal Suppose du is removed from student slice Di..." but the corresponding data point's 's removal affects a *different* teacher constituent model Ty. where (k, 1) * (p, v). In this case, two separate unlcarning processes occur concurrently:
1. Student constituent model Sy handles the removal of d, using the standard SISA process: revert to Sk..--1 and retrain onwards.
2. Student constituent model S,, handles the update resulting from teacher Ta, unlcarning its data, following the PURGE process for teacher unlearning: revert S, and retrain onwards using updated soft labels.
The total unlcarning time is approximately the sum of the times for these two independent partial retraining processes. In both scenarios, PURGE requires only partial retraining of at most two student constituent models. This contrasts sharply with a naive SISA application, where any teacher update (as occurs in both scenarios) would necessitate full retraining of all N student constituent models. Therefore, PURGE offers substantial efficiency gains even when handling simultaneous unlearning requests.
A.4 Training Stability: Multi- Melti-Teacher vs. Single-Teacher Soft Labels Approaches
Our experimental results demonstrate that the incremental multi-teacher learning strategy introduced by the PURGE framework consistently outperforms the single-teacher soft label ablation approach. This advantage primarily stems from the enhanced training stability provided by multi-teacher soft label generation. Unlike conventional epoch- based methods, SISA-based unlearning frameworks employ incremental training over data slices, enabling checkpointing at the point each data slice first introduced. While this design allows efficient backtracking to specific checkpoints for unlearning requests, introduces potential instability. Specifically, early training rounds on smaller data slices may cause the model to overfit to those initial slices. Subsequently, when new slice is added, the
22

---

model's loss can fluctuate more than in standard epoch- epoch-based training, since it must train on both previously learned and new data, resulting in greater gradient variation and increased instability.
Loss Across Cumulative Chunk Training Steps
The PURGE framework introduces additional data chunk partitioning to enable precise map- ping between teacher and student constituent models. This design exacerbates instability in the single-teacher soft-label setup: the model must simultaneously adapt to both newly intro- duced data and an unfamiliar teacher constituent model responsible for generating the correspond- ing soft labels. To illustrate this effect, Figure 4 plots the loss curve during the training of stu- dent constituent model, showing the dynamics when learning from a shard comprised of data chunks with soft labels produced by 8 teacher constituent models.
single-Tracher Mult-TTacchrr
20
Cumulative Chunk Training Steps
Figure 4: Loss curve comparison between multi- teacher soft label generation and single-teacher ablation
The blue curve in Figure corresponds to the single-teacher ablation while the orange curve being the proposed multi-teacher soft label generation. We plot the loss against the cumulative chunk training steps, counting every instance chunk is trained (including repeats). The vertical dashed lines are the indicators when new data chunk is introduced to the incremental training process. Except for the first interval, which involves training on a single chunk, each subsequent chunk training step selects a different chunk for training. This process cycles through all available chunks turn, ensuring that each chunk is trained for exactly eR epochs before the next chunk is introduced.
From the plot, we observe that during the first three intervals, the loss curves for the single-teacher ablation and the multi-teache sobt-label training are quite similar, as the training process cycles through only a small number of data chunks. However, as training progresses and more chunks are introduced, the single-teacher ablation exhibits significantly larger fluctuations. This increased instability arises because the model alternates between data chunks that have already been trained extensively and newly added chunks. The newly introduced chunks not only provide previously unseen inputs, but their corresponding soft labels generated by newly introduced teacher model are also unfamiliar to the student. This unfamiliarity can result in pronounced changes in the loss and, consequently, larger gradients, leading to an unstable training process.
In contrast, our multi-teacher soft-label generation approach mitigates such fluctuations. Although the data may still be new to the student model, the soft labels are produced by averaging predictions across multiple teacher models from prior training steps. This aggregation moderates sharp changes and smooths the training dynamics. As a result, while some fluctuations are still present, their magnitude is much lower than in the single-teacher ablation, providing more stable training process and ultimately better model performance.
A.5 Performance on distillation with smaller student training dataset
In knowledge distillation, it is common for the student be trained on smaller subset of the data distilled by the teacher. With less overall training data, each student constituent model will receive smaller data allocation. Consequently, the balance between increasing the number of constituent models for faster unlearning and ensuring each constituent model is well-trained becomes more critical. To investigate how the proposed method may perform under such conditions, we conduct experiments on the 10%, 20%, and 50% subsets of MNIST and SVHN. The experimental results show that PURGE produces similar performance to SISA under all conditions, with more stable performance compared to Single-teacher Soft Label when each student constituent model learning from large number of teacher constituent models. Overall, the effectiveness of PURGE demonstrated.
23

---

nt 8
nt 16
nf 32
MNIST
99
99
Accuracy /%
Accuracy %%
98
Accuracy /
98
97
07
97
10%
96
96
8
12
16
16
24
Number Students
Number of Students
Number Students
99
39
99
Accuracy %
Accuracy %
Accuracy
97
97
20%
96
80
2
300
8
12
16
24 Number Students
16
32
Number Students
Number of Students
99
99
99
Accuracy 1%
Accuracy %
Accuracy
97
07
97
60%
30
95
95
16
95
24 Number of Students
16
Number of Students
Number of Students
SVHN
90
90
Accuracy %
Accuracy %
Accuracy %
85
85
80
80
10%
73
70
70
12
16
24 Number Students
16
Number Students
Number of Students
90
30
90
Accuracy %
Accuracy %
Accuracy %
8
80
80
80
20%
25
70
2
70
9
12
16
70
24 Number Students
16
32
Number Students
Number of Students
90
90
90
Accuracy %
&
Accuracy
Accuracy
80
80
&
60%
PURGE
20g steche 2ott Label
70
70
16
70
16
24
Number Students
Number of Students
Number of Students
Figure 5: Comparison of student network accuracy on 10%, 20% and 50% versions of MNIST and SVHN. The plot shows results for PURGE, the SISA baseline student, the original Teacher ensemble, and the Single-teacher Soft Label ablation. The original Teacher ensemble was trained on the full training sets, while the student networks are trained on the corresponding subsets.
24

---

nt
nt 16
nt 32
50
50
Accuracy
Accuracy
Accuracy
-30
30
80
CIFAR-100
20
20
00
10
Number
Students
55
Accuracy 1%
Accuracy 1%
Accuracy 1%
50
SSTS
45
400
400
24 Number of Students
16
Number of Students
Number of Students
Accuracy 16
Accuracy %
Accuracy %%
50
2815
BERT
45
DESA
PURSE
Waggetosce
400
16
24
Number of Students
Number of Students
Number of Students
Figure 6: Comparison of student network accuracy on CIFAR-100 (top row) and SST5 (mid and bottom rows). For the SST5 experiment, employed both BERT and Qwen2.5-7B models, distilling their knowledge into BERT student models. Accuracy is plotted against the number of student constituent models (N) for different teacher ensemble sizes (M = 8, 16, 32). The plot shows results for PURGE, the SISA baseline student, the original Teacher ensemble, and the Single- teacher Soft Label ablation. Models were trained the full training sets of CIFARO- 1000 and SST5.
teacher
A.6 Experimental Results: CIFAR-100, SST-5, and Large Language Models (Qwen2.5 7B/32B)
We evaluated performance on the more challenging CIFAR-100 dataset for image classification using ResNet50 and the SST5 dataset for sentiment analysis using BERT. Figure shows the performance of PURGE compared to the basclines under these conditions.
The complexity of the image classification task on CIFAR-100 is reflected in the relatively low accuracy of the Teacher ensemble itself, which achieved 30.5% on the test set with a teacher ensemble of M 32 constituent models. Despite the task's difficulty and the performance gap relative to the original Teacher, PURGE demonstrates effectiveness by achieving performance very comparable to the SISA baseline student across different numbers of student constituent models (N), As seen in Figure 6 the accuracy of PURGE closely follows that of the SISA baseline while significantly outperforms the Single-teucher Soft Labels, particularly when the number of students small. When the number of students is large, especially when the teacher and student have the same number constituent models, M = N,, we see the proposed PURGE outperforms bascline SISA.
This highlights that for complex tasks, the constituent mapping introduced in the proposed PURGE framework enables each student constituent model to learn fine-grained information contained in the soft labels produced by its corresponding teacher sub-ensemble of constituent models. In contrast, the baseline SISA aggregates teacher outputs, which can obscure these detailed signals and lead to reduced performance. The results also highlight the broader challenges of ensemble learning in multi-teacher distillation, with such challenges also existing for the proposed PURGE framework since soft-label aggregation still occurs within each teacher sub-ensemble. A promising direction for future work to explore more refined multi-teachher aggregation strategies while preserving the data isolation between teacher constituent models established in PURGE.
25

---

Although sentiment analysis is different task from image classification, the results on the SSTS dataset shown in Figure exxibit similar trend to those observed in image-based tasks, In this experiment, we trained two versions of the teacher model: one based on BERT and the other on Qwen2.5-7B. Both teacher models were trained for e' = 20 epochs, and in both cases, distilled to BERT student models. Interestingly, the results of the BERT and Qwen-based experiments are highly comparable, with the more capable Qwen2.5-7BB teacher delivering only modest improvements over the BERT teacher. This minor performance difference suggests that the limiting factor may be the sharding strategy used in the SISA framework, rather than architectural differences or model scale. Notably, the largest difference between the two scenarios arises in the results for the single- teacher ablation, where Qwen2.5- didttlllle students outperform their based counterparts, demonstrating that singlettacher setups may experience larger performance drops, even when the teacher model itself differs slightly. By contrast, the proposed PURGE framework maintains stable performance across both version of the experiment. In both cases, PURGE demonstrates performance comparable to SISA, while the incremental multi-teacherr distillation approach proves more effective than using angle-teacher soft labels alone. Combined with the results on CIFAR-I00 these findings confirm PURGE's viability for moderately complex tasks, its scalablility to larger model like Qwen2.55 7B, and its general applicability across different domains, preserving the accuracy of standard SISA distillation while introducing substantial unlearning efficiency.
To further extend our evaluation beyond sentiment analysis, applied the PURGE framework to the more demanding domain of language reasoning on the ARC dataset employing Qwen2.5-32B as the teacher model and distilling into Qwen2.5-3B student models. Using 8-teacher, 8-student configuration and LoRA [16] fine-tuning, we compared PURGE with naive SISA on ARC-challenge four-choice questions. In this scenario, the teacher only achieved 27. 27.76% accuracy, and both PURGE and naive-SISA exhibited limited student performance, 26. .9% and 26.42% accuracy, respectively. It reflects the further constraints imposed by sharding strategies, which limits the data availability for each constituent model, in complex reasoning tasks. As a result, even the teacher merely surpasses chance levels, and student models trained under both frameworks struggle to provide high accuracy. These findings underscore a pressing need for future work to achieve stronger results while maintaining strict data isolation required for sharding- based verified unlearning.
Despite these accuracy limitations, our analysis of unlcarning efficiency highlights the practical value of PURGE. Leveraging an 8 Nvidia A100 GPU setup, the PURGE framework enabled student-side updates in just 2.36 GPU-hours for teacher-side unlearning requests, compared to 18.9 GPU-hours for nAS..-SISA. Given that each teacher checkpoint occupies 128GB and each student checkpoint uses 12GB, PURGE does require storage of multiple checkpoints for each student constituent model. For example, in our experimental setup with eight student constituent models, each composed of one chunk and four slices per chunk, the overall storage requirement for student checkpoints reached 386GB. In similar distillation settings, this additional storage demand for student models remains well within the capabilities of typical server infrastructures. The trade-off between increased storage requirements and accelerated unlearning can be flexibly managed according to the specific demands of the application, taking into account available storage resources and the expected frequency of unlearning requests. More importantly, PURGE achieves a substantial reduction computational time and cost for verified unlearning in large-scale deployments, outperforming both naive-SISA A and conventional retraining from scratch. Thus, even as scaling challenges remain for complex tasks, the PURGE framework demonstrates significant improvements in computational resource efficiency for unlearning, positioning it as a practical approach for efficient, privacy-preservin model deployment.
A.7
Data Mapping
Data mapping is a critical component of sharding-basee unlcarning frameworks. In this work, we use uniform random partitioning as it provides generality, prevents systematic bias, and scales well by enabling uniform hyperparameters across constituent models.
While uniform partitioning is the default, alternative strategies are possible. Notably, distribution- aware sharding, as demonstrated in the original SISA framework, groups data points that are more likely to be unlearned into dedicated shards. This can optimize unlearning speed in scenarios where such prior knowledge is available and our PURGE framework is fully compatible with such alternatives.
26

---

In addition to distribution aware sharding, our study of the student-teacher knowledge distillation framework also raises important questions regarding how data is mapped between students and teach- ers, particularly when faced with imbalanced data distributions. To investigate this, we conducted experiments on the SST-5 dataset focusing on label-imbalanced partitioning. In this setting, each teacher shard was deliberately structured to have one class overrepresented and the others under- represented. with a fixed class ratio of 0.24:0. 10..11900111000 19:0.19:0.11 19 across the five sentiment categories. For each of the five teacher shards, one class was assigned the largest proportion (0.24) while the rest were each allocated 0.19, and the final shard contained any remaining samples cover the whole training dataset.
Table 2: Impact of iebeliimbllnneed partitioning on teacher and student performance under matched and mismatched data mappings on SST-5, The reported values are the model accuracy.
No. of students
1
2
4
8
Matched (Single)
45.02%
48.57%
49.59%
49.77%
Matched (PURGE)
50.59%
51.02%
51.87%
49.77%
Mismatched (Single)
45.10%
48.78%
49.59%
25.34%
Mismatched (PURGE)
50.72%
50.90%
51.9 51.95%
25.34%
Teacher Baseline
48.86%
We explored two main scenarios: a "matched" setup, where each student was trained on a data shard exhibiting the same class imbalance as its corresponding teacher, and "mismatched" configuration. where students received shards with different imbalanced class distributions. In both scenarios, we trained the models using our proposed PURGE framework with the multi-teache soft label strategy and, for comparison, the single-teacher soft label ablation. Each approach utilized eight teacher constituent models and varied the number of student models. Both the teacher and student models are implemented using BERT. Our results (Table demonstrate that under matched conditions. the accuracy of teacher models fell from 51.32% (with uniform partitioning shown in Figure 6) to 48.86% with the imbalanced class ratio. This decline was even more pronounced following 1-to-1 distillation, with student accuracy sharply dropping 25.34% by training student constituent models, each paired with single teacher. This significant degradation highlights how even relatively mild label imbalances in partitioning can be strongly amplified by the distillation process, particularly in small datasets like SST-5, where each shard contains approximately 1,000 examples.
However, the impact of label imbalance is substantially mitigated once we move away from the strict matching between teacher and student distributions. Training each student on a mismatched data chunk led significant recovery in performance, with student accuracy rising to 48.77%. Importantly, once students learn from multiple teachers, the performance gap between matched and mismatched scenarios becomes minimal for both single- and multi-teacher soft label generation setups. Furthermore, aggregating soft labels from multiple teachers further enhanced resilience to bias. For instance, in the cight-teacher, four-student configuration, mismatched student accuracy reached 51.95%. closely matching the 51.84% achieved under matched conditions, and both outperformed the single-teacher ablation baselines (49.68% and 49.59% for mismatched and matched, respectively). These trends validate the robustness of incremental multi-teacher distillation, which consistently outperforms single-teacher setups and maintains data isolation between teacher shards.
The results show that while label imbalance can significantly amplify bias and degrade performance under strict matching, this effect is mitigated by mismatched mappings or by aggregating information from multiple teacher constituent models. Consequently, the choice of partitioning strategy could be guided by the application context and available prior knowledge, with uniform random partitioning serving as a robust default and alternative strategies offering potential efficiency gains and maintain performance when appropriately applied.
A.8 Broader Impacts
Artificial intelligence has become deeply integrated into many aspects of society, with generative models, large language models (LL.Ms), and other AI tools significantly boosting societal productivity. However, training these models, particularly L.MMs, can incur substantial costs. For instance, Sam Altman stated that the cost of training GPT-4 was more than $100 million.
27

---

Beyond financial cost, the environmental impact is also significant. Luccioni et al. [23 estimated the training process of BLOOM, 176B parameter language model, contributed around 24.7 tonnes of carbon dioxide equivalent (COzeq) emissions. Therefore, reducing unnecessary model training brings both economic and environmental benefits.
Verified unlearning methods are designed to safeguard user data privacy by ensuring that once data removal requested, all influence of that data effectively and provably eliminated from the trained model. This strict compliance with data regulations like GDPR and CCPA, which grant individuals the right to request their personal data be erased, requires more than simply deleting data from the training set. It requires the guarantee that the model no longer retains any information derived from the removed data. By enabling efficient model updates without the need for full retraining, verified unlearning safeguards user data privacy, meets the rigorous legal standards, but also provides clear economic and environmental benefits in the process, particularly in real-world scenarios where large volume of unlearning requests may occur.
Our proposed PURGE framework is specifically designed for verified unlearning in the context of knowledge distillation (KD), which is widely used in domain adaptation and model compression for local deployment of machine learning models, e.g., the distillation of models like ChatGPT. Existing verified unlearning methods typically require full retraining of the student model whenever the teacher model is updated, due to the lack of solutions tailored the student- teacce paradigm. PURGE addresses this gap by enabling efficient model updates in response to unlearning requests on either the teacher or student side. As KD is common practice, the proposed PURGE framework can significantly reduce the economic cost and alleviate the environmental impact associated with the retraining process. Thereby, the proposed PURGE can contribute to more sustainable and privacy- compliant Al deployment, bringing broader societal impact through responsible and efficient use of machine learning technology.
28


Reviews:
[{"review_id": "Z0lnkdnE79", "reviewer": "Reviewer_guLt", "summary": "This paper proposes PURGE, a framework for efficient and verified machine unlearning in the context of knowledge distillation (KD). While existing methods like SISA provide provable unlearning in single model settings, They don't work well when the teacher changes, because the updates spread through the whole model and require retraining all the students.\n\nTo address above issues, PURGE introduce constituent mapping to isolate teacher influence to specific student constituents and also apply incremental multi-teacher distillation, where student constituents are trained in stages using subsets of their assigned teacher ensemble, preserving data isolation.\n\nThe authors provide both theoretical analysis (showing ≥N× speedup for N student constituents) and empirical validation across several datasets (MNIST, SVHN, CIFAR-100, SST5), showing that PURGE substantially improves unlearning efficiency without degrading predictive performance.", "questions": "Please see the questions below, which are related to the weaknesses mentioned:\n\n1. Are there any more realistic scenarios that could be considered? For example, the original SISA paper explores transfer learning from a pre-trained ImageNet model. Would a similar setting help validate the method in a more practical context?\n\n2. Could additional experiments be designed to provide deeper insights into the strengths and limitations of the proposed method?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a novel problem setup by identifying a specific failure case for SISA when applied in the context of knowledge distillation.\n\n- The proposed method is clearly explained, and the logic is easy to follow, making it accessible for readers.\n\nWeaknesses:\n\n- While verified unlearning in image and text classification is important, the scope of the paper feels somewhat narrow. It only considers relatively small models like ResNet-50 and BERT. This makes the claim about transferring knowledge from \"large, computationally intensive teacher models\" feel less convincing. It would be stronger to include experiments involving large reasoning models being distilled into smaller LLMs with reasoning abilities (Just one example I can image). \n\n- The data mapping process might introduce bias depending on the models used. How is the mapping strategy chosen? Are there any optimal ways to split data that could help improve or degrade performance? A deeper discussion would be helpful.\n\n- The analysis experiments are limited. While the method does seem to address the specific failure of SISA under knowledge distillation, the paper would benefit from additional experiments designed to provide more insights into the strengths and limitations of the proposed approach.", "was_revised": true, "final_justification": "The additional experiments provided during the rebuttal have largely addressed my concerns regarding the narrow scope, demonstrating broader applicability within the presented setting.\n\nThe discussion on data sharding was helpful and clarified the methodological choices.\n\nHowever, the work still primarily focuses on classification evaluations, which limits the generality of the conclusions.\n\nOverall, the rebuttal has resolved most of my main concerns, and I have updated my recommendation to borderline accept.", "rebuttal": "## Narrow scope of models\n\nWe thank the reviewer for their constructive feedback on the scope of models evaluated. We agree that demonstrating PURGE's effectiveness on larger, more contemporary models strengthens our paper's claims.\n\nOur original choice of models was a deliberate decision driven by the computational demands of our experimental protocol. To rigorously validate our framework, our protocol tests numerous configurations of teacher (M) and student (N) constituents, requiring the training of entire ensembles from scratch for each data point shown in our performance graphs. Using established and still-realistic models like BERT and ResNet-50  was therefore necessary to conduct this thorough, multi-run validation, which would have been intractable with larger models during the initial research phase.\n\nHowever, to directly address the reviewer's valuable point, we use the rebuttal period to run new, targeted experiments on large language models (LLMs). We have completed one LLM-scale experiment and have a second, even larger run currently in progress. The results below validate that PURGE's efficiency gains extend to modern, multi-billion parameter distillation pipelines. \n\nDuring the rebuttal period, we tested PURGE with 8 teachers and 8 students, distilling Qwen-2.5 7B into BERT-base on SST-5. Due to SST-5's small size (8,854 samples), we froze the pretrained Qwen model's base and fine-tuned only the classification head to avoid overfitting. The teacher achieved 51.4% validation accuracy, while PURGE reached 50.7% (comparable to naive SISA's 50.9%) with a 7.7× speed-up, despite increased soft-label generation costs.\n\n| Experiment Status | Teacher → Student | Dataset | Test Accuracy| Verified Unlearning Time† | Speed-up vs. Naïve‡ |\n| --- | --- | --- | --- | --- | --- |\n| **Completed** | Qwen-2.5 7B → BERT-base | SST-5 | 50.7% | **0.17 GPU-hours** | **7.7×** |\n| **In-Progress** | Qwen-2.5 32B → Qwen-2.5 3B | ARC-Challenge | Coming soon | ***est. 2.36 GPU-hours** | ***~8 x (18.9 GPU-hours)** |\n\n† Wall-clock time for a single constituent to unlearn one chunk, reflecting an unlearning request affecting one teacher.\n‡ Naïve baseline refers to the full retraining of the student network required when a teacher is updated without the PURGE framework.\n*Estimated through dummy runs involving soft-label from Qwen-2.5 32B (0.06 GPU-hours) and training the Qwen-2.5 3B model (2.3 GPU hours) on data chunks with dummy teachers with an 8-teacher-8-student setup. \n\nOur new experiments confirm PURGE delivers efficient, verified unlearning at scale. These results align with our theoretical analysis: PURGE's efficiency comes from its structure and is model-agnostic. As shown in Equations 3 & 4, speed-up depends only on student constituents (N) and chunks (c), not teacher model size. This means larger teacher models yield greater absolute time and computational cost savings. These LLM experiments validate that our guarantees hold at scale.\n\nTo fully substantiate these findings, we launched the 32B and the 7B parameter experiment on an 8xA100-80GB node immediately upon receiving the reviews. We will include the completed learning curves and further details in the camera-ready paper, providing a comprehensive validation of PURGE's scalability and directly addressing the reviewer's comments.\n\n## Data mapping process\n\nWe thank the reviewer for their insightful question about data mapping strategies. This is a crucial aspect of sharding-based unlearning frameworks. Our random partitioning approach was deliberately chosen, and we agree that discussing alternatives would be valuable. \n\nIn our paper, we use **uniform random partitioning**, the standard approach from SISA, for three main reasons: (a) **Generality:** It makes no assumptions about data distribution or future unlearning requests; (b) **Bias prevention:** It ensures each constituent receives a representative data sample; and (c) **Scalability:** It allows using consistent hyperparameters across all constituent models. \n\nThe reviewer correctly notes that non-uniform splits are possible. The original SISA paper demonstrates **distribution-aware sharding** where data points likely to be unlearned are isolated into dedicated shards. This approach optimizes unlearning speed but reduces generality. While not explored in our paper, PURGE is compatible with these strategies. Our core contribution, constituent mapping and incremental distillation, works independently of the initial data mapping. PURGE could be combined with distribution-aware partitioning for enhanced efficiency when prior knowledge exists. Our final paper will discuss this more thoroughly.\n\nIn addition to distribution-aware sharding, we conducted a study on **label imbalance** in data partitioning using SST-5. We compared **uniform random partitioning** with **label-imbalanced partitioning**, where each teacher shard has one overrepresented class (e.g., first shard overrepresents 'very positive', second shard overrepresents 'positive'). Shards follow a class ratio of 0.24:0.19:0.19:0.19:0.19, with the final shard containing remaining samples. The experiment used 8 teacher shards.\nWe examine two scenarios:\n\n- **Matched:** Each student is trained on data with the same class distribution as its corresponding teacher shard.\n- **Mismatched:** Each student is trained on a different, imbalanced data chunk than the corresponding teacher.\n\nThe immediate effect of class imbalance is a drop in teacher model performance. The accuracy falls to **48.86%**, compared to **51.32%** in the uniform partitioning case. This degradation is amplified in the **matched 8-student** setup, where student accuracy drops to **25.34%**, highlighting how even mild label imbalances can be magnified through distillation, especially in small datasets like SST-5, where each shard contains ~1000 examples.\n\nHowever, when we move away from this extreme matched condition, either by training students on **mismatched** shards or by aggregating outputs from multiple teachers, student model performance improves substantially. For example:\n\n- In the mismatched 8-student setup, accuracy rises to **48.77%**.\n- In the 8-teacher, 4-student setting:\n    - **Mismatched:** student accuracy reaches **51.95%**\n    - **Matched:** student accuracy is **51.84%**\n\nIn both cases, our **PURGE** method outperforms the **single-teacher ablation**, which yields **49.68%** (mismatched) and **49.59%** (matched) accuracy.\n\n### Key findings from these results:\n\n1. When teacher and student share the same label bias (**$c=1$ matched**), student performance can degrade significantly due to amplified bias.\n2. Even with **$c=1$**, training on a **mismatched** bias can substantially mitigate this degradation.\n3. Learning and aggregating soft labels from **multiple teachers** helps alleviate the effect of bias even further. Our **incremental multi-teacher distillation strategy** consistently outperforms single-teacher ablation while maintaining data isolation between teacher shards.\n\nThese are part of our preliminary results. We commit to including a full analysis of these results in the final version of the paper. This will expand on the trade-offs between partitioning strategies, offering guidance on when different approaches might be warranted and opening avenues for future research into optimizing data mapping specifically for unlearning in knowledge distillation.\n\nWe believe this combination of clarifying our principled default choice, discussing the existence of optimal strategies, and presenting the findings from new experiments will fully address the reviewer's valuable feedback.\n\n## Coverage of experimental results\n\nOur initial experimental design was intended to establish the foundational viability and generality of the PURGE framework across a wide range of settings. Our experiments already provide several key insights: (a) Our **speed evaluation** demonstrates that PURGE's unlearning efficiency scales predictably with the number of student constituents (N), providing empirical validation for our theoretical speed-up analysis; (b) The **ablation studies** confirm that our incremental multi-teacher distillation strategy is critical for maintaining student performance, showing significant advantages over a single-teacher approach and thus validating a core design choice of our framework; (c) By testing on diverse datasets (MNIST, SVHN, CIFAR-100, SST-5) and model architectures (CNNs, ResNet-50, BERT), we show that PURGE's principles are not confined to a single domain but are broadly applicable.\n\nThat said, we agree that more targeted analysis, especially involving large, modern models, provides a clearer picture of the framework's strengths. To address this, we have conducted experiments with multi-billion parameter Qwen models as well as the evaluation on label imbalance conditions. These experiments offer deeper insights into the practical application of PURGE. \n\nThese additional results provide a more nuanced understanding of PURGE's strengths. We will integrate this deeper analysis into the final version of the paper to fully address the reviewer's comments.\n\n## Response to Questions\nAs most points are addressed above, we provide a brief summary below due to space constraints:\n1. The distillation paradigm itself, where a student learns from a teacher trained on a large dataset, is conceptually very similar to transfer learning; the student inherits knowledge without access to the original, large-scale training data. However, to more directly address this point, we have conducted **new experiments with LLMs** as described above.\n2. We agree that providing deeper insights is crucial. Our initial experiments were designed to establish the foundational principles and general applicability of PURGE. During rebuttal, we have included the following experiments to provide a deeper analysis.\n    1. **Large-model distillation**\n    2. **Data mapping strategy**"}, {"review_id": "ir3h3T3yAX", "reviewer": "Reviewer_w4Vk", "summary": "The paper addresses a significant and timely problem (efficient verified unlearning in KD, specifically for teacher updates) with a novel and well-motivated framework (PURGE). The core idea (constituent mapping + incremental multi-teacher distillation) is sound and effectively tackles the critical information propagation issue. Theoretical analysis and thorough experiments demonstrate clear efficiency gains while maintaining accuracy.", "questions": "1. The analysis shows speedup decreases as c (chunks/student) increases for fixed M. The paper suggests using N=M (c=1) for max speedup. However, increasing N linearly increases the total number of student constituents to train and store, plus the overhead of aggregating N predictions during inference. The trade-off between speedup and total resource consumption (compute, memory, storage) needs explicit discussion. Is N=M always practical?\n\n2. The description of the incremental training process (Section 3.1, steps 1-3, and the state transitions S_{k,l,j}) is dense and slightly hard to follow on first read. A small pseudocode snippet could significantly improve clarity.\n\n3. The trade-off regarding r (slices per chunk) is mentioned (Sec 3.2 end, Sec 4.1) but its impact on student-side unlearning efficiency isn't quantified. How much faster is student unlearning with larger r? This is important context for choosing r.\n\n4. The term \"constituent\" is used heavily. While defined, consistently using \"model constituent\" or \"submodel\" initially might help readability.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Tackling verified unlearning specifically for teacher-side updates in KD is a crucial and previously under-addressed challenge.\n2.  The central idea of constituent mapping and the incremental multi-teacher distillation strategy is elegant, well-motivated, and effectively solves the information leakage problem inherent in naive SISA application to KD.\n3.  Experiments are comprehensive, covering multiple datasets (image, text) and tasks. The core claims are well-supported.\n4. The solution makes verified unlearning practically viable for teacher-student pipelines, which is essential for deploying and maintaining large models under privacy regulations.\n\nWeaknesses\n- The theoretical speedup Nx is a minimum under idealized assumptions (uniform sharding, uniform slices/chunks, linear scaling). Real-world overheads (checkpointing I/O, varying slice sizes, non-linear training dynamics) are ignored. The ceiling function impact is noted but its practical significance for large rc isn't deeply explored.\n\n- Soft Label Regeneration Cost: The efficiency analysis (Fig 2, Eq 1-4) explicitly excludes the time to regenerate soft labels (Y_{k,i} for i >= l) after a teacher update. This cost, especially for large models or large chunks, could be substantial and potentially dominant over the partial student retraining time in PURGE. While the paper argues inference cost is \"negligible,\" this is highly dataset/model-dependent and needs quantification, particularly for the large models mentioned. Comparing PURGE's total time (teacher update + soft label regen + student partial retrain) vs. naive SISA's total time (teacher update + student full retrain) is crucial for a fair efficiency assessment. This is a significant weakness in the current efficiency claim.\n\n- While the focus is on verified unlearning, comparing PURGE's efficiency/accuracy trade-off against state-of-the-art approximate KD unlearning methods (even if only student-side focused or lacking guarantees) would provide valuable context. How much efficiency is sacrificed for the verification guarantee?\n\n- PURGE requires storing intermediate student checkpoints per student constituent. For large models and many slices/chunks, this storage overhead could be prohibitive. This is not discussed.", "was_revised": true, "final_justification": "The authors provided clear and detailed responses to the key concerns, including empirical validation of theoretical speed-up, soft label regeneration cost, storage-compute trade-offs, and practical implications of design choices. These clarifications resolved the main issues I raised.  I maintain my score of weak accept.", "rebuttal": "**On the idealised theoretical speed-up.**  We thank the reviewer for this important point. We agree that our theoretical analysis operates under idealized assumptions, which is a standard approach used to isolate and clarify the core architectural benefits of a new method.\n\nThe most crucial validation, however, comes from our empirical results. As shown in our speed evaluation (Figure 2), the **measured, real-world speed-up** of PURGE on the MNIST dataset **closely tracks our theoretical predictions**. This demonstrates that while real-world overheads from I/O and non-linear dynamics exist, they do not dominate the process or invalidate the fundamental efficiency gains our framework provides.\n\nIt is also important to note that many of these overheads, such as checkpointing I/O, are inherent to any SISA-based verified unlearning framework, including the naive SISA baseline. Therefore, they do not disproportionately affect the *relative* speed-up that PURGE achieves.\n\n**On the cost of soft label regeneration.** This is a very important point, and we thank the reviewer for pushing for more detail. We acknowledge that excluding this cost from our primary analysis was an oversimplification, as the time required is non-zero and model-dependent.\n\nTo provide the quantification requested, we analysed the costs in our new **Qwen-2.5 7B experiment with an 8-teacher-8-student setup on the SST-5 dataset.** Regenerating soft labels for affected data chunks took ~32 seconds on an A100 GPU, while partial student retraining took ~620 seconds. Soft label regeneration thus accounts for just ~5.2% of total student-side unlearning time despite using a large model. This small proportion exists because inference needs only a single forward pass per data point, while retraining requires multiple epochs of forward and backward passes plus optimizer steps. This asymmetry increases with model size. Importantly, PURGE's total unlearning time remains much lower than the baseline. In naive SISA, updating the teacher requires regenerating soft labels for the entire training set, which is far more expensive than PURGE's partial regeneration. The fair comparison is:\n\n- **PURGE Total:** (Teacher Retrain) + (**Partial** Soft Label Regen) + (**Partial** Student Retrain)\n- **Naïve SISA Total:** (Teacher Retrain) + (**Full** Soft Label Regen) + (**Full** Student Retrain)\n\nGiven that partial student retraining is orders of magnitude faster than full retraining, the overall efficiency claim holds true even when accounting for the soft label regeneration cost. We will clarify this in the final version.\n\n**On comparison to approximate unlearning methods.**  We agree that contextualising our work against approximate methods is useful. As we discuss in our Related Work (Section 2), verified and approximate unlearning represent two distinct paradigms with different goals and guarantees.\n\n- **Approximate methods** offer speed and flexibility but provide no formal, provable guarantee of complete data removal.\n- **Verified methods**, like PURGE, provide a formal/verifiable guarantee that the post-unlearning model is from the identical probability distribution as a model trained without the data from the start.\n\nThis distinction is critical for regulatory compliance (e.g., GDPR's \"Right to be Forgotten\"), where a provable, auditable removal of influence may be a legal requirement. In such high-assurance contexts, the trade-off is not about sacrificing a small amount of efficiency for a guarantee; rather, approximate methods are often not a viable alternative.\n\nBecause our work is situated in the verified unlearning paradigm, the most relevant baseline is the only other verified exact approach: full model retraining. Our contribution is to make verified unlearning in KD pipelines dramatically more efficient than this baseline.\n\n**On the storage overhead of checkpoints.**  The reviewer correctly identifies that storing intermediate checkpoints incurs a storage cost, which can be significant for large models. This is an inherent trade-off for all SISA-style verified unlearning methods. However, this storage cost must be weighed against the computational cost it saves. In our **in-progress Qwen2.5-32B parameter experiment**, while the checkpoints require substantial disk space, they allow us to reduce an unlearning operation from an estimated 18.9 **GPU-hours** (for full retraining) to just 2.36 **GPU-hours, even with a modest 8-teacher-8-student setup**.\n\nIn modern ML infrastructure, the cost of disk storage is typically orders of magnitude lower than the cost of high-end GPU compute time. Sacrificing cheaper storage to save enormous amounts of expensive compute is a highly practical and accepted engineering trade-off. Furthermore, as noted in the original SISA paper, large-scale training pipelines already employ frequent checkpointing to prevent data loss from hardware failures. PURGE's slicing mechanism can often be integrated into these existing checkpointing practices with minimal additional overhead. We will add a discussion of this practical trade-off to the final paper.\n\n\n## Response to questions:\n\n1. **On the practicality and resource trade-offs of the `N=M` configuration.**  Thanks for this important practical question. We agree that while the $N=M$ (i.e., $c=1$) configuration provides the theoretical maximum speed-up for teacher-side unlearning, it is not always the most practical choice when considering total resource consumption. The optimal configuration involves a direct trade-off that a system designer must make based on their specific priorities. The key trade-offs are: \n\n    a. **Increasing `N`** (the number of student constituents): This **maximises the unlearning speed-up**. However, it also linearly **increases initial training costs, storage requirements**, and **inference latency** (due to aggregating more models). Furthermore, as shown in our performance evaluations (Figures 3, 5), making $N$ too large can degrade model accuracy by leaving each constituent with too little training data. This claim is supported by our extended experiments addressing the effect of data mapping and class imbalance in shards (see Reviewer guLt’s comments). We introduced additional evaluations on the SST-5 dataset, where both teacher shards and student chunks were intentionally imbalanced. The full details of these experiments will be included in the Appendix of the camera-ready version of the paper. The key findings from these results include: \n\n    - When teacher and student share the same label bias (**$c=1$ matched**), student performance can degrade significantly due to amplified bias. \n    - Even with **$c=1$**, training on a **mismatched** bias can substantially mitigate this degradation.\n    - Learning and aggregating soft labels from **multiple teachers** helps alleviate the effect of bias even further. Our **incremental multi-teacher distillation strategy** consistently outperforms single-teacher ablation while maintaining data isolation between teacher shards. \n\n    These results emphasize the advantage of using larger student shards (i.e., $c > 1$) under these extreme conditions. Thus, the optimal configuration for $N$ is a balance between unlearning speed, overall system costs and model performance.\n\n\n    b. **Decreasing `N`** (and thus increasing $c$, the chunks per student): This **reduces the overhead** of training, storage, and inference, and can **improve model accuracy**. The trade-off is a **slower unlearning time** for teacher-side updates.\n\n    Therefore, the choice of $N$ is task-dependent. Our paper provides the experimental data on performance vs. $N$ to help practitioners make this informed decision. We will add an explicit discussion of this trade-off to the final version.\n    \n2. **On the clarity of the Incremental training process.**  We thank you for the constructive suggestion. We agree that the description of our incremental training process is dense and that a pseudocode snippet would significantly improve its clarity. We will add a detailed pseudocode algorithm to the appendix in the camera-ready version of the paper to explicitly detail the training loop for a single student constituent.\n\n3. **On quantifying the `r` trade-off for student-side unlearning.**  This is an excellent point, and we thank you for asking for this quantification. Student-side unlearning in PURGE follows the standard SISA procedure. Therefore, its efficiency is determined by the total number of slices per shard, $R$. In our framework, $R$ is the product of chunks ($c$) and slices-per-chunk ($r$), i.e., $R=c \\cdot r$. The analysis from the original SISA paper shows that the expected unlearning cost for a single request is proportional to $(\\frac{2}{3} + \\frac{1}{3R})$ times the cost of retraining the full shard. This means that as $r$ increases, $R$ increases, and the unlearning time for a student-side request **decreases**, approaching a maximum speed-up of **1.5x** (compared to a shard with no slicing). This clarifies the trade-off:\n    1. **Increasing $r$:** Improves student-side unlearning speed.\n    2. **Decreasing $r$:** Improves teacher-side unlearning speed, as shown in our paper's Equation 4.\n\nThe optimal choice of $r$ depends on the expected frequency of student-side vs. teacher-side unlearning requests for a given application. We will add this explicit quantification to the final version.\n\n4. **On the readability of the term \"constituent\":**  Thanks for this helpful suggestion to improve readability. This is an easy and valuable fix. In the camera-ready version of the paper, we will revise the text to use **\"constituent model\"** or **\"sub-model\"** in place of \"constituent\" where appropriate, particularly in the early sections, to ensure the terminology is immediately clear to the reader."}, {"review_id": "iswasYPmJL", "reviewer": "Reviewer_PBdg", "summary": "The paper proposes PURGE, an extension of the data-isolation verified-unlearning framework SISA to knowledge distillation (KD). SISA’s core idea is to shard and slice data so only affected slices retrain when deletion requests arrive. In naive KD, however, shard-isolated teachers become interdependent when soft labels are computed using all the teacher models, re-coupling the shards and potentially leaking deleted information.\n\nPURGE restores isolation by applying sharding and slicing to both student data and teacher ensemble. Specifically, it introduces a chunk-and-slice schedule: within each shard, teachers are grouped into overlapping sets, tied to disjoint chunks; the student is distilled incrementally, starting with soft labels from a single-teacher + chunk and progressively moving through additional data-chunks with more and more teachers being used in additional chunks for soft-label generation, till all the teachers are used. Each chunk of student data are further sliced, enabling checkpoint-based rollback exactly like SISA. When a teacher or data point must be forgotten, only the student slices whose training relied on chunks containing that teacher --- or on the deleted data --- require regeneration, reducing unlearning cost while preserving KD accuracy. Experiments show significant wall-clock speedups without accuracy loss.", "questions": "1. Could you report an ablation where $c = 1$ (no chunking) but the number of teacher constituents is increased so that each student still maps to exactly one teacher?\nThis would separate the benefit of data-isolation itself from the chunking-over-time schedule.\n\n2. You claim incremental multi-teacher distillation yields “a more stable training process.” Could you provide convergence plots (loss/accuracy vs steps) or variance metrics to substantiate this?\n\n3. Does constituent mapping still give verified guarantees when the output space is unbounded (e.g., sequence generation)? If so, how would you adapt PURGE to autoregressive decoding?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The authors identify an important problem of unlearning in the KD setup, and the extension of SISA is natural and well explained. The clever bit is mapping teacher slices to student chunks so only a slice-of-a-student retrains when a teacher unlearns; Nice, even though incremental.\n- Experiments span a good scale of models from small models on simple MNIST datsets to models with a few 100 million parameters at BERT scale.\n- The paper is well written and conveys its ideas cleanly.\n\n**Weaknesses**\n\n- KD-per-shard baseline under-explored. A straightforward alternative is one teacher–student pair per shard. Even if inference latency rises, memory traffic, activation size, and communication overhead could make this baseline competitive on larger datasets (e.g., ImageNet-1k).\n\n- Chunk-free ablation. The paper would benefit from deeper analysis into the chunk-free ablation. Set $c = 1$ and increase the number of shards/teachers until total teachers = M. Report (a) unlearning latency and (b) accuracy. This isolates the benefit of chunking itself. Consider the chunk-free baseline where,  `#shards` $\\leftarrow$ `# chunks * # shards`. Here, the number of student models increase by `#chunks`, but no ordered training is required at the chunk level and each student works with smaller amount of data. Would this not make unlearning more efficient and the process more resource efficient?\n\n- Does the method admit any easy extension to open-domain tasks?\n\n- Stability claim: Incremental multi-teacher \"provides a more stable training process\" will benefit from substantiation beyond accuracy curves. For instance, including loss/accuracy traces or gradient-norm stats across multiple training runs/models with other aspects kept constant would be beneficial.", "was_revised": true, "final_justification": "After carefully considering the authors’ rebuttal and the discussion among reviewers, I have decided to retain my original score of 4: Borderline accept. The authors provided thoughtful clarifications and additional insights that helped address several of the initial concerns and miss-understandings. Overall, the paper presents a technically sound contribution with a promising idea that could be valuable to the community, after the promised additions are included.", "rebuttal": "**On the KD-per-shard baseline.**  We thank the reviewer for proposing this interesting alternative architecture. A \"KD-per-shard\" setup, with parallel and independent teacher-student pairs, is indeed a valid approach for distributed learning and simplifies the unlearning process.\n\nHowever, our work is specifically designed to address the unlearning challenge within the **ensemble knowledge distillation** paradigm. This paradigm is widely used precisely because aggregating knowledge from an ensemble of diverse teachers often produces a single, stronger student model than any individual teacher could. The KD-per-shard baseline fundamentally changes this setup into a collection of independent, single-teacher distillations, thereby losing the performance benefits of ensembled knowledge.\n\nPURGE is designed to **preserve the accuracy benefits of ensemble distillation** while making it compatible with the rigorous demands of verified unlearning. While a direct empirical comparison between these two different distillation paradigms is a valuable direction for future research, the focus of our paper was to solve the unlearning problem within the established and powerful ensemble distillation setting.\n\n**On the chunk-free ($c=1$) ablation.** The \"chunk-free\" or $c=1$ scenario corresponds precisely to the $N=M$ configuration in our experiments, where each student constituent is mapped to exactly one teacher constituent. Our existing experiments in Figures 3 and 5 already explore this configuration and its implications. As our theoretical analysis also shows (Equation 4), this $c=1$ case provides the **theoretical maximum unlearning speed-up** for teacher-side updates.\n\nHowever, this comes at the cost of **maximum resource consumption and potentially lower accuracy**. The $N=M$ configuration requires training the largest number of student models, leading to the highest initial training costs, storage overhead, and inference latency. Most importantly, it often results in lower final accuracy because each student is trained on the smallest possible data shard ($1/N$) while learning from one teacher only. In our extended experiments addressing the effect of data mapping and class imbalance in shards (see Reviewer guLt’s comments), we introduced additional evaluations on the SST-5 dataset, where both teacher shards and student chunks were intentionally imbalanced. The full details of these experiments will be included in the Appendix of the camera-ready version of the paper. These results further emphasize the advantage of using larger student shards (i.e., $c > 1$). Under the extreme conditions we simulated, we observed that the negative impact of class imbalance can be significantly mitigated once each student begins learning from multiple teachers. Therefore, while the $c=1$ ablation is the fastest for unlearning, it is not always the most practical choice. The optimal configuration is a balance between unlearning speed, overall system costs and model performance, a trade-off our paper provides the data to analyse.\n\n**On the extension to open-domain tasks.** Yes, the PURGE framework is designed to be model- and task-agnostic and admits easy extension to open-domain tasks like sequence generation. The core unlearning mechanism of PURGE operates on the *training structure* (data partitioning, constituent mapping, and checkpointing), not on the specific model architecture or loss function. To adapt PURGE for an autoregressive task, one would:\n\n- Use sequence-to-sequence models (e.g., transformers) for the teacher and student constituents.\n- Employ a sequence-level distillation loss (e.g., minimising the KL divergence over the vocabulary distribution at each time-step).\n\nThe fundamental unlearning logic would remain unchanged. If a teacher constituent $\\mathcal{T}_{k,l}$ that contributed knowledge from chunk $l $ onwards is unlearned, the student still `$\\mathcal{S}_k$` reverts to the checkpoint `$\\mathcal{S}_{k,l-1}$` and retrains from there with updated soft labels. Our new experiment on the **ARC-Chllenge reasoning task** is a step in this direction, demonstrating PURGE's flexibility beyond simple classification.\n\n**On substantiating the \"stability\" claim.**  This is a valuable suggestion. We agree that providing more direct evidence for the improved training stability of our incremental multi-teacher approach will strengthen the paper. The performance drop of the \"Single-teacher Soft Label\" ablation in our experiments is a strong indicator of this instability, as the supervisory signal changes abruptly between chunks. To substantiate this claim more directly, as requested, **we will add convergence plots to the Appendix** in the camera-ready version.  To substantiate this claim more directly, we made the convergence plots for the training processes of PURGE and “Single-teacher” ablation. These plots will be added to the Appendix in the camera-ready version. We expect these traces will visually demonstrate a smoother and more stable convergence for our proposed method, providing more explicit evidence for our claim.\n\n## Answers to questions:\n\n1. **On the chunk-free ($c=1$) ablation.**  Our existing experiments already explore this \"chunk-free\" ($c=1$) scenario. This case corresponds to the configuration where the number of student constituents equals the number of teacher constituents ($N=M$), which is an endpoint on our experimental plots (e.g., Figures 3 and 5). Our analysis shows that this $c=1$ configuration provides the **theoretical maximum unlearning speed-up** for teacher-side updates. However, this speed comes at the cost of potentially lower model accuracy, as each student constituent is trained on the smallest possible data shard ($1/N$). The benefit of chunking ($c>1$) is therefore not to improve unlearning speed, but rather to **improve final model accuracy and reduce resource overhead** by allowing each student to be trained on a larger data shard while still learning from a diverse ensemble of teachers.\n2. **On substantiating the \"stability\" claim.** The significant performance degradation of the \"Single-teacher Soft Label\" ablation, particularly when each student learns from many teachers sequentially (i.e., when $c$ is large), is strong evidence of the instability caused by abrupt changes in the supervisory signal. To substantiate this claim more directly, as requested, **we made the convergence plots and will add them to the Appendix** in the camera-ready version of the paper. These plots will compare the training loss and validation accuracy curves over training steps for our incremental multi-teacher approach (PURGE) versus the \"Single-teacher\" ablation. These plots can visually demonstrate a smoother and more stable convergence for our proposed method, providing more explicit evidence for our claim.\n3. **On extension to open-domain tasks.**  Yes, the constituent mapping provides the same verified guarantees for open-domain tasks with unbounded output spaces, such as sequence generation. This is because the PURGE framework is **model- and task-agnostic**; its unlearning guarantee is a property of the training architecture (isolation, checkpointing, and reversion), not the specific model or loss function. Adapting PURGE to an autoregressive task would be straightforward: (a) the teacher and student constituents would be sequence-to-sequence models like transformers; (b) the distillation objective would be a sequence-level loss, such as minimising the KL divergence between the teacher's and student's output distributions at each decoding step. The core unlearning mechanism would **remain identical**. If a teacher constituent $\\mathcal{T}_{k,l}$ requires unlearning, the corresponding student `$\\mathcal{S}_k$` still reverts to the checkpoint `$\\mathcal{S}_{k, l-1}$` created before that teacher's influence was introduced. This process is independent of the task. Our experiment on the SST5 sentiment analysis task using BERT models is a step in this direction, demonstrating the framework's applicability to modern Transformer architectures."}, {"review_id": "MBEBQv0prx", "reviewer": "Reviewer_zLhe", "summary": "This paper proposes PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), an unlearning framework tailored for knowledge distillation. The method proposes a student-teacher constituent mapping and an incremental multi-teacher distillation strategy to ensure data isolation during the distillation process. The approach extends SISA to allow partial student retraining when teacher-side unlearning occurs, avoiding the inefficiencies of full student retraining. Experiments demonstrate that PURGE achieves significant speed-ups in unlearning while maintaining performance.", "questions": "1. Machine unlearning refers to removing the influence of specific training data from a trained model. In this work, the student-side unlearning operations are more accurately characterized as \"cascading synchronization/update after unlearning\" rather than unlearning. This distinction is conceptually important. Could the authors clarify how they define unlearning and what distinguishes the proposed approach from standard SISA beyond the cascading synchronization mechanism?\n\n2. The term \"verified unlearning\" is used repeatedly throughout the paper. Could the authors provide a precise definition of what constitutes \"verified unlearning\" in this context, along with the formal criteria or standardized references that support this terminology? Additionally, what theoretical proofs or empirical guarantees demonstrate that the resulting model is equivalent to one trained without the deleted data, thereby justifying the \"verified\" claim?\n\n\n3. The paper assumes SISA as the default verified unlearning framework without questioning its scalability limitations, such as high training cost from maintaining multiple models and performance degradation due to data fragmentation. Is the sharding architecture truly necessary for machine unlearning, or could alternative approaches achieve similar goals without these inherent overhead costs?\n\n4. Have the authors considered the practical utility of approximate unlearning methods, and could they provide a comparative analysis of the trade-offs between exact and approximate approaches in terms of performance, efficiency, and regulatory compliance?", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality：\nThe method is sound and builds on SISA, but it inherits SISA’s scalability issues, maintaining many models and checkpoints is costly. Experiments are limited to small datasets, with no substantial evaluation on large models like BERT.\n\n\nClarity: The writing is clear and well-organized. Key ideas like constituent mapping and incremental distillation are explained logically and are easy to follow.\n\nSignificance:\nThe paper defines the student-side retraining as part of machine unlearning, but this is conceptually questionable. Machine unlearning aims to remove training data influence from a trained model, whereas the student model here never directly sees the data being unlearned. The proposed method is better viewed as a pipeline update mechanism, NOT unlearning.\n\nOriginality:\nAlthough the paper explores a new angle in distillation, it does not introduce a fundamentally new unlearning method, as it is entirely based on SISA. Since the student model does not directly unlearn any data, the work falls outside the core definition of machine unlearning.", "was_revised": true, "final_justification": "Due to insufficient contributions, I will maintain my score.\n\n1) The use of SISA to teacher and student models is overly simplistic. The contribution is limited to a mapping between teacher and student models.\n\n2) The understanding Verified Unlearning are fundamental misunderstanding. No verification process/effort was found in the claimed \"Verified Unlearning\".  \n\n3) The work inherits all limitations of SISA.", "rebuttal": "We thank the reviewer for their detailed and thought-provoking feedback. You have raised several important conceptual questions that go to the heart of how unlearning is defined in complex systems like knowledge distillation. We are grateful for the opportunity to clarify our work's formal guarantees and demonstrate its significance as a necessary advancement in the verified unlearning landscape.\n\n**Apologies for any inconsistency between mathematical notation and LaTeX expressions here. Some symbols may not render correctly.**\n\n**On the definition of unlearning in a distillation pipeline.** We thank the reviewer for this nuanced question. We would like to clarify why we firmly believe this work falls within the scope of machine unlearning. The core issue, as we state in our introduction, is that \"information about the teacher's training data can leak and propagate pervasively throughout the student network\"  during distillation. Even if a data point $d$ is perfectly unlearned from a teacher $\\mathcal{T}$, its influence may persist in any student $\\mathcal{S}$ already trained by $\\mathcal{T}$. Our work addresses the critical and non-trivial task of verifiably removing this leaked influence from the student.\n\nThis process is more rigorous than a \"cascading synchronisation.\" A simple update might involve just swapping the teacher model. Our method, in contrast, adheres to the core principle of SISA-based unlearning: when a teacher constituent $\\mathcal{T}_{k,l}$ is updated, the student $\\mathcal{S}_k$ is reverted to a checkpoint `$\\mathcal{S}_{k,l-1}$` that was saved *before* any influence from `$\\mathcal{T}_{k,l}$` was introduced. This rewind-and-retrain procedure is a fundamental unlearning operation, not merely a pipeline synchronisation.\n\nTo formalise this, let $\\mathcal{A}$ be the training algorithm. A teacher $\\mathcal{T}$ is trained on its data $\\mathcal{D}^T$, so $\\mathcal{T} = \\mathcal{A}(\\mathcal{D}^T)$. A student $\\mathcal{S}$ is then trained via distillation, making it a function of the teacher's outputs and its own data $\\mathcal{D}^S$: $\\mathcal{S} = \\mathcal{A}(\\mathcal{T}(\\mathcal{D}^S), \\mathcal{D}^S)$. When a point $d$ is removed from $\\mathcal{D}^T$, the teacher is updated to $\\mathcal{T}' = \\mathcal{A}(\\mathcal{D}^T \\setminus d)$. The goal of student-side unlearning is to produce a new student $\\mathcal{S}'$ such that $\\mathcal{S}' = \\mathcal{A}(\\mathcal{T}'(\\mathcal{D}^S), \\mathcal{D}^S)$.\n\nThis ensures the final student $\\mathcal{S}'$ is in a state equivalent to having been trained from the start by a teacher that had never seen $d$. This removal of $d$'s indirect influence is the **central unlearning challenge we solve**.\n\n**On the meaning and proof of “verified unlearning”.**  We use the term \"verified unlearning\" consistent with its definition in the SISA literature. It provides a formal guarantee that the resulting model is computationally indistinguishable from a model trained from scratch on the dataset with the specified data point removed; this exclusion process can be verified due to the framework’s engineered design. More formally, the post-unlearning model is drawn from the identical probability distribution as a model trained on the redacted data from the start.\n\nThis guarantee is achieved **by construction**, and our algorithm itself serves as the proof. PURGE ensures this in the distillation context as follows: (1) **Isolation:** The framework maintains strict data isolation, ensuring a teacher constituent's influence is confined to its mapped student constituent; (2) **Reversion:** To unlearn data affecting teacher $\\mathcal{T}_{k,l}$, the student `$\\mathcal{S}_k$` is reverted to the checkpoint `$\\mathcal{S}_{k,l-1}$.`  (3) **Untainted state:** This checkpoint `$\\mathcal{S}_{k,l-1}$` is, by construction, provably untainted by the data to be unlearned, as it was created before that data's influence (via teacher $\\mathcal{T}_{k,l}$) was ever introduced to the student; (4) **Retraining:** All subsequent training proceeds from this clean state using the updated teacher, $\\mathcal{T}'_{k,l}$.\n\nTherefore, the final student model is guaranteed to be from the same distribution as one trained without the influence of the removed data, satisfying the \"verified\" claim.\n\n**On SISA’s limitations and the necessity of sharding. **The reviewer correctly identifies that sharded architectures like SISA involve overhead in terms of model and checkpoint maintenance. We acknowledge this known trade-off in the field. Our work's contribution is not to solve SISA's inherent limitations, but to make SISA's verified unlearning guarantees **possible** in the ubiquitous knowledge distillation paradigm, where a naive application fails and forces costly full student retraining. \n\nFor **verified unlearning**, data isolation is a prerequisite to avoid full retraining in this model-agnostic setting. Without an architectural mechanism like sharding, the influence of a single training point propagates throughout the entire model during training. Surgically removing this influence without affecting the contributions of other data points is not possible, making full retraining the only other 'verified' alternative. The overhead of sharding, while real, is the price for achieving fast, provable unlearning, and as our new LLM experiments show, this cost is vastly lower than the alternative of full model retraining for large-scale models.\n\n**On comparison with approximate unlearning methods.** We agree that positioning our work relative to approximate unlearning is important. As we discuss in our Related Work (Section 2), verified and approximate unlearning are two distinct paradigms with different objectives. Approximate methods often rely on heuristics to estimate and reverse a data point's contribution. They are often faster but provide no formal guarantees of complete removal, which may be insufficient for contexts with strict legal or privacy requirements, such as GDPR's \"Right to be Forgotten\".\n\nOur work is situated squarely in the **verified unlearning** paradigm. Its goal is to provide a formal, provable guarantee of data removal. Therefore, the most relevant baseline for our efficiency claims is the only other verified approach available in this setting: full model retraining. Our experiments demonstrate that PURGE is orders of magnitude more efficient than this baseline. We believe this focus on providing a verified solution to a problem where none previously existed is a significant contribution.\n\nWe hope these clarifications address the reviewer's concerns and underscore the significance of our work in extending provable unlearning to the critical domain of knowledge distillation.\n\nAnswers to questions:\n\n1. **On the definition of unlearning in a distillation pipeline.** Machine unlearning is defined by the removal of a data point's **influence**. In knowledge distillation, the student is indirectly influenced by the teacher's training data. Our work addresses the necessary task of removing this leaked influence from the student model. This process is distinguished from a simple \"cascading update\" by its core mechanism, which is inherited from SISA: we revert the student to a provably untainted checkpoint from *before* the influence was introduced and then retrain. This \"rewind-and-retrain\" process is a fundamental unlearning operation.\n2. **On the meaning and proof of “verified unlearning”.** We use \"verified unlearning\" consistent with its definition in the SISA literature: the post-unlearning model is drawn from the **identical probability distribution** as a model trained from scratch on the redacted data; the data removal process can be verified thanks to the framework’s design. This guarantee is not empirical but is achieved **by construction**. The algorithm itself is the proof: by reverting to a checkpoint that is provably untainted by the data to be removed and retraining from that clean state, the final model is guaranteed to be from the correct distribution.\n3. **On SISA's limitations and the necessity of sharding.** We acknowledge that SISA-based frameworks involve an overhead trade-off. Our paper's contribution is not to solve SISA's inherent limitations but to make its guarantees **possible** in the ubiquitous knowledge distillation paradigm, where a naive application fails. For **verified** exact unlearning, data isolation via an architectural choice like sharding is a prerequisite to avoid full retraining, which is the only other verified alternative. Our new LLM experiments confirm the overhead of our approach is far less than that of full retraining.\n4. **On comparison with approximate unlearning methods.** As discussed in our Related Work (Section 2), verified and approximate unlearning are two distinct paradigms. Approximate methods offer speed but lack the formal, provable guarantees of removal that may be required for regulatory compliance (e.g., GDPR). Our work is situated in the **verified unlearning paradigm**. Therefore, the most relevant baseline is the only other verified approach: full model retraining. Our experiments show PURGE is significantly more efficient than this baseline, and our contribution is providing a verified solution where one did not previously exist."}]

Author rebuttal:

Meta-review: This paper studies exact unlearning for knowledge distillation settings where unlearning requests may refer to either the data that was used to train the teacher or the data used to train the student. In this setting, if SISA is used on the student model, then student-side unlearning requests can be supported exactly and efficiently, but the system would still require full student retraining if teacher-side unlearning requests occur. To address this, the authors propose an extension of SISA for knowledge distillation settings called PURGE. PURGE is a multi-teacher distillation framework designed to isolate the influence of each teacher to specific student constituents. It comes with a special training regime where students are trained in stages using subsets of their assigned teachers. By construction, the algorithm ensures data isolation and enables exact unlearning for both teacher-side and student-side requests that is more efficient than naively training the system from scratch. The authors offer theoretical analysis quantifying the speed-ups achieved by their method over naive retraining from scratch. They also offer empirical results on various datasets (further expanded during the rebuttal to include larger language models) where they demonstrate that their approach obtains significant speedups without sacrificing the student’s predictive performance.

The reviewers noted that the problem setting studied is novel, important and under-addressed, the proposed method is sound, well-motivated, clearly explained and effective. Some reviewers found the experimental scope too narrow initially but the authors (partially) addressed this concern through larger-scale experiments during the rebuttal (though limited to classification tasks). Reviewers raised other concerns about the limited justification of the mapping strategy used in PURGE, the cost of soft label regeneration not being taken into account for efficiency calculation, the need for additional ablations, and the scope of the work and placement within the broader literature, that were sufficiently addressed during the rebuttal via clarifications and discussions.

The key weaknesses raised by the reviewers are the scalability issue of PURGE (inherited from SISA) due to having to store many models and checkpoints, and the limited novelty of the proposed method, which can be viewed as being somewhat incremental over SISA, rather than proposing a new algorithm. One reviewer raised a concern about referring to PURGE as “verified unlearning”. The authors have clarified their use of the term (which is valid under the assumption of an honest service provider that the authors will state in the revised manuscript). 

Generally, while the PURGE does have some limitations (requires storage of checkpoints), it is the first practically viable exact unlearning method for teacher-student unlearning that supports both teacher- and student- side unlearning requests more efficiently than naive retraining; a problem that reviewers recognized is practically important and understudied. I feel like this is a compelling reason to accept the paper which trumps concerns raised about other aspects like current scalability limitations (since it's the first existing solution for this new setting). The proposed method is sound and has novel components that were not previously proposed (a specialized distillation procedure that preserves data isolation, going above and beyond simply applying SISA twice). Based on this, I view it as a significant step forward that warrants acceptance.
Decision: Accept (poster)