Unlearning Evaluation through Subset Statistical Independence
Authors: Chenhao Zhang
Muxing Li
Feng Liu
Weitong Chen
Miao Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Machine Unlearning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Evaluating machine unlearning remains challenging, as existing methods typically require retraining reference models or performing membership inference attacks, both of which rely on prior access to training configuration or supervision labels, making them impractical in realistic scenarios. Motivated by the fact that most unlearning algorithms remove a small, random subset of the training data, we propose a subset-level evaluation framework based on statistical independence. Specifically, we design a tailored use of the Hilbert–Schmidt Independence Criterion to assess whether the model outputs on a given subset exhibit statistical dependence, without requiring model retraining or auxiliary classifiers. Our method provides a simple, standalone evaluation procedure that aligns with unlearning workflows. Extensive experiments demonstrate that our approach reliably distinguishes in-training from out-of-training subsets and clearly differentiates unlearning effectiveness, even when existing evaluations fall short.

Published as a conference paper at ICLR 2026
UNLEARNING EVALUATION
THROUGH SUBSET STATISTICAL INDEPENDENCE
Chenhao 2 Muxing Li,, Feng Liu Weitong Chen', Miao Xu University of Queensland, 2 University of Melbourne, Adelaide University,
*RIKEN Center for Advanced Intelligence Project
{chenhao. zhang, miao. xu)euq edu au, muxing. 11 Li@student. wts.tudeet..uimmellbb.... wwwmeln.mm.......... eew.ll..euua... au feng. liulllllle @unimell edu au, t. chen@adelaide, edu. au
ABSTRACT
Evaluating machine unlearning remains challenging, as existing methods typically require retraining reference models or performing membership inference attacks, both of which rely on prior access to training configuration or supervision labels, making them impractical in realistic scenarios. Motivated by the fact that most unlearning algorithms remove small, random subset of the training data, we propose a subset-level evaluation framework based on statistical independence. Specifically, design a tailored use of the Hilbert-Schmidt Independence Crite- rion to assess whether the model outputs on a given subset exhibit statistical de- pendence, without requiring model retraining auxiliary classifiers. Our method provides a simple, standalone evaluation procedure that aligns with unlearning workflows. Extensive experiments demonstrate that our approach reliably dis- tinguishes in-training from out-of-training subsets and clearly differentiates un- learning effectiveness, even when existing evaluations fall short. The codes are available at https: http//gttttt com/ Chi IdDden/SDE.
INTRODUCTION
Machine Unlearning aims remove the influence of specific training data samples from trained model. This capability is crucial for both adversarial-oriented unlearning, where backdoors or cor- rupted knowledge introduced during training must be eliminated, and privacy-oriented unlearning, where individuals may request their data be erased under data protection regulations such as the "right to be forgotten" (Garg al., 2020; Bukaty. 2019). To meet this need, recent works (Nguyen et al., 2022; Xu et al., 2024) have developed various unlearning algorithms to eliminate the influence of subset of training data that needs be forgotten, i.c., forgetting data, from trained model.
One key challenge lies in verifying whether an unlearning process has been successful, especially in realistic deployment settings where retraining from scratch impractical. Existing unlearning works often assess unlearning effectiveness by comparing metrics such model utility (e.g., ac- curacy) (Shen et al., 2024; Fan et al., 2024) and retraining time (Tarun et al., 2024; Zhang et al., 2024) against a retrained model. In these evaluations, the closer the unlearned model resembles the retrained model, the better its unlearning effecctivnness. Nevertheless, this evaluation paradigm suffers from major limitation: it relies access to a retrained model trained with remaining data only, which defeats the purpose of developing a standalone, verifiably unlearned model.
Membership inference attacks (MIA) are often used to evaluate unlearning by testing whether spe- cific sample was seen during training. Existing MIAs rely on three main cues: 1) confidence scores, assuming models assign higher confidence training samples (Salem al., 2019); loss-based criteria, using the empirical gap in training and held-out loss (Yang et al., 2016); and 3) auxiliary classifiers trained on prediction vectors or hidden representations (Shokri al., 2017). These meth- ods require access to internal training statistics (c.g., loss distributions, confidence ranges) and often rely on shadow models trained with the same data distribution or hyperparameters. Such assump- tions rarely hold in post-hoc unlearning evaluation, where the original training setup or sufficient
"Co-correspondence.

---

Published as a conference paper at ICLR 2026
data is unavailable (Chundawat et al., 2023), making it infeasible to reconstruct loss baselines or train effective attacker models. Moreover, unlearning methods are typically required to remove a small, random subset of the training data (5%-20%) (Nguyen et al., 2022; Xu et al., 2024), creating two practical challenges: 1) limited data sample or label provide too few supervision to reliably train auxiliary classifiers, and 2) per-sample cues like loss confidence become statistically weak after unlearning because the subset loses co-adaptation with the remaining data during unlearning. In this context, pursuing accurate general per-sample inference inefficient and misaligned with the un- learning workflows. Instead, what matters is whether the subset as whole retains any statistically detectable signal of prior training. We therefore propose a shift from sample-wise MIA to subset- level evaluation, where we test statistical dependence among the model outputs candidate for- getting set. Our motivation stems from that training participation induces inter-sample dependencies in the model's ss internal representations due to shared gradient updates and co-adaptation. In contrast, for data never seen during training, such inter-sample dependency should not arise.
In this work, we propose Split-half Dependence Evaluation (SDE) that evaluates the effectiveness of unlearning by determining whether a given subset is in-training data based on statistical dependence among the model's outputs of the subset. Specifically, we adopt the Filbeer-Schhidt Independence Criterion (HSIC) (Gretton et al., 2005a; 2007), a widely used kernel- measure well-suited for high-dimensional data, and we novelly propose the split-half dependence test where subset is split into two halves, and the dependence between their activations is computed via HSIC. Our analysis shows that the split-half dependence test catches the inner subset dependence with shared sample influence component introduced by the model's training. Unlike existing unlearning evaluations, our method: 1) enables unlearning evaluation without needing retrained reference model: 2) does not rely on auxiliary classifiers or additional model training, and 3) operates on data subsets rather than individual samples, resulting in a simpler and more robust evaluation that better aligns with the overarching goal of unlearning. Extensive experiments on the retrained models demonstrate that our method can effectively identify the in- and out-of-trainngg subsets. Experiments on existing unlearning methods demonstrate that our method can verify unlearning success even in settings where existing evaluations struggle to provide conclusive evidence.
Due to space constraints, we discuss related works-including unlearning evaluation, MIA, and sta- tistical independence-in Appendix B.
2 PRELIMINARIES
Let Dir denote the original training dataset, and Die denote the test dataset. Let h represent a neural network model. Given an input X, the h(x) is the final layer activation. Deep neural networks may consist of many layers, so we use he(#) E Rdim to denote the activation from the f-th layer with the dimension of dim. Specifically, we use (x) to denote the activation from the penultimate layer, since it is often used as the extracted feature of the input.
2.1 MACHINE UNLEARNING
In the context of machine unlearning, the forgetting data, D, C Dir: the subset of the training data whose influence is intended to be removed. Correspondingly, the remaining data is denoted as D, = Dir Dr. Given a deep neural network model with a specific architecture, we consider three variants of the model in the context of unlearning. The original model o is trained on D... The unlearned model un is obtained by applying an unlearning algorithm to remove the influence of D, The retrained model he is special unlearned model, which is trained on D, from scratch, is usually used as the gold standard.
2.2 HILBERT-SCHHE SCHMITT INDEPENDENCE CRITERION (HSIC)
HSIC is a kernel-based statistical measure that quantifies the degree of dependence between two random variables. Given two random variables X and Y, HSIC(X, is the cross-covariance operator between the reproducing kernel Hilbert spaces (RKHS) of X and Y. and || IæS denotes the Hilbert-Schmide norm. HSIC value is non-negative and continuous, providing a meaningful scale: the closer HSIC is to zero, the more independent the two variables are; higher values indicate stronger statistical dependence.
|Cxy li where CXY
2

---

Published as a conference paper at ICLR 2026
Empirically, the X and Y can be two sets of observations with the same sample size, i..., |X| | Given kernel functions defined over the respective domains of the variables, the empirical estimator of HSIC can be expressed as
1 Tr(KHLH), (n 1)2
HXIC
where K and I. are the kernel matrices for and Y, respectively, and is the centering matrix H = - 117.. Following (Gretton et al., 2007), use the Gaussian RBF kernel as the default choice throughout this paper.
3
METHOD
Our main motivation stems from when is the result of a supervised training algorithm .A and trained on Dur ie., h = A(Dr.), the learned parameters inherently depend the training data. To catch such dependence, a naive idea would be to directly compute the dependence between network parameters and training data, e.g., HSIC h). However, this is problematic because has only one observation, as there is usually only one trained network, and it consists of millions of parameters, making statistically unreliable and computationally prohibitive.
Instead, we consider the dependence among h(x:)'s. Intuitively, we treat the deep neural network h(-) as complex transformation from the input space the output space. If h consists of randomly initialized parameters, i.e., random transformation, then the outputs h(x;;) and h h(22,) ) for any two samples ztz F E Dir should remain independent. When h A(D,.). hxx;) implicitly depends on 2i through the learned parameters; hence, the outputs l((;;) and h(xz)) are no longer indepen- dent. In contrast, out-of-training samples (i.e., Dic) are not involved shaping the parameters of h. thus their activations should exhibit weaker statistical dependence. Motivated by this, as well as by the fact that unlearning typically forgets subset of the training data, we introduce the Split-half Dependence Evaluation (SDE). Given target subset, we split it into two random halves and mea- sure the dependence between their representations. In later subsections, we empirically validate that under our split-half evaluation, in in-training subsets exhibit greater dependence than out-of-trainnng subsets. Appendix A presents an analysis showing that a shared, training-induced influence com- ponent yields higher split-half dependence for in-training subsets, which can be reflected by toy experiment in the Appendix A.4. The overall algorithm is presented in Appendix C.
3.1 SPLIT-HALF DEPENDENCE OF SET OF DATA
Given a target set of data and model h, we want to measure the dependence between the activa- tions of samples in S on h. Specifically, we randomly divide S into two equal sets, S₁ and Sg, and propose the Split-half Dependence H(S, h) by
H(S, h) (ISCC(((S),,SS)))
where S₁ U2 S1 n 0, and |S|| |S2). Since HSIC is statistical metric, we follow the practice of Gretton et al. (2007) to shuffle Sy 200 times to calculate 200 HSIC values for estimating the (S, h) distribution. As motivated earlier, the dependence of in-training data (STT C Der) activations on a trained model her should be significantly higher than that of out-of-training data (Soor C Dtc), i.e., (Srr, h) > H(Soor,A h).
We illustrate this difference by plotting the distributions of H(Srr, h) and H (SSoor. h) using both the trained model A and the randomly initialized model h rand As shown in Figure 1, the split-half dependence of the in-training subset SrT under the trained model oor is significantly higher than that of the -ftt--nin subset SooT- In contrast, under the randomly initialized model h rand the two distributions largely overlap, suggesting distinguishable dependence signal.
To demonstrate the statistical significance, we further conduct a one-sided Mann-Whitee U- test (Mann & Whitney, 1947), under the alternative hypothesis of (ST,, > H[SoOT, h). This U-test is chosen because it is non- parammeeee and directly tests our alternative hypothesis H(Srr, h) > H(SooT, h) with established statistical rigor. We reject the null hypothesis and accept this alternative if the p valuu satisfies p < 0.01. As shown in Figure 1, the split-half dependence dis- tributions under the trained model or differ substantially, with p-value of 4.73 10-11 & 0.01,
3

---

Published as a conference paper at ICLR 2026
HiSpor
Mision Aran)
Mi hey MiSoor
M Sun
pm4.75e11
p=7.95e022
Disenn Sour. Mour--bh 10....000000000 827
D1 Son hm) 008
Sig Sor A 110 .088 p- 4.73e-11
01 oo (10 pa 1.600-112
Figure 1: Empirical (S,,)) distributions cal- culated in-training subset Sir and out-of- training subset Soot using two models: (left) trained model hor and (right) randomly initial- ized model raaadd
Figure 2: H h) distributions on the origi- nal model o (left) and the retrained model he (right). The Star is in-training for hor but out- of-training for hT" In the retrained model, the Star becomes significantly closer to the SooT than the Syr in terms of HSIC
indicating that the in-training subsets exhibit significantly higher dependence than out-of-training subsets. contrast, under the randomly initialized model h htand, the distributions largely overlap, yielding non-significant p-value of 0.0795.
3.2 EVALUATING THE UNLFARNED MODEL VIA H(),h)
Given the empirical observation that the (S,, h) distributions diverge significantly between in- training and out-of-trainnng data under trained model, we exploit this property assess the efficacy of an unlearning process in forgetting a specific subset from the original training set.
given target subset from its forgetting data, i.e., Star Dr, and aim to determine whether it resembles in-training or out-of-training data from the unlearned model's perspective. To this end, reserve small subset of known in-training data SIT C D. and out-of-training data Soor C Die. referred to as the reference sets. In practice, reference sets can be constructed from small portion of the training and test data that inten- tionally retained for auditing or debugging purposes. We first obtain all three S's split-half depen- dence distributions, i.e., H(S, hum), and then compare (S hmm) to the reference distributions H (SpT, hnm) and H (SoOT, h. If H (Starr h=m) is significantly closer to (SOOT hmm), we infer that the model exhibits behavior consistent with being trained without Star- Conversely, if is closer to H (SIT, h,n), we conclude that the nu is trained with Star- For presentation convenience, we define D(SA, SB. h) to measure the distance between the split-half dependence distributions two subsets SA and Sg under model h. Therefore, an unlearning is considered successful if
When evaluating unlearned model hmn,
DSStar; SooT, 1" < (SSarr, Syr, h=m), Star S Dj.
(2)
We adopt the Jensen-Shannon Divergence (JSD) (Fuglede & Topsøe, 2004) compare the split- half dependence distributions due to its favorable properties in our context. Compared to alternatives such as KL divergence and Wasserstein distance, JSD is symmetric, bounded, and numerically stable for empirical distributions with overlapping support. The choice of JSD simplifies the design of an efficient Algorithm and avoids potential instability issues. Specifically, if the JSD applied,
SB, h) JSD (H h) h)).
We conduct experiments with the or and the retrained ht shown in Figure 2, the distribu- tion of (Star, ht is closer that of the Soot than to the Syr. yielding ((Star,Son) ht)) < D(Star SIT, )))). contrast, when evaluated under the original model har trained on the full dataset, the H (Star hor) remains closer the ((Sr, hor). This supports that Eq. 2 provides practical signal for detecting whether a group of samples was present the model's training data.
4 EXPERIMENT
This section is organized as follows: (1) We begin by conducting controlled experiments on re- trained models, where confirm that forgetting data is not involved in the model's training pro- cess. This allows us to verify that our proposed method can effectively distinguish between in- and out-of-training subsets, and to investigate its robustness across different conditions from aspects of
4

---

Published as a conference paper at ICLR 2026
model architectures, dataset scales, and representation layers. (2) We compare the proposed statisti- cal independence-bur based method with commonly used distribution based metrics. (3) We then apply it to evaluate widely-use -usedy unlearning baselines. This enables to compare their unlearning effec- tiveness in unified way. A computational cost analysis and corresponding experiment are included in the Appendix G.
4.1 CONTROLLED EXPERIMENTS ON RETRAINED MODELS
We conduct experiments on four benchmark datasets: SVHN (Netzer et al., 2011), CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and Tmny-ImageNer. We train two neural network classifiers, the AlICNN (Springenberg et al., 2015) and ResNet-18 (He et al., 2016). The dataset configurations and model architectures are summarized in Appendix D. Appendix E presents case study applying our method to evaluate a diffusion generative model. For every lataset-model setting, we train 5 re- trained models with different random seeds
the reliability of our results. We consider four key aspects: (1) Kernel bandwidth a: HSIC is kernel-based dependence measure, the kernel band- width plays crucial role in accurately estimating statistical dependence. (2) Impact of [2y] and |S]: How does the size of the subset S and the proportion of forgetting data, i.c., R = [Drl/|Dur] affect the method's ability detect it? (3) Layer-level robustness: Does the method remain valid when evaluating features from internal layers (hę) instead of just the last layer? (4) Training-stage robustness: Can our method reliably assess unlearning for models at different points in training, e.g., across different epochs? These questions help establish the practical utility and robustness of our method under various realistic scenarios, before applying it to unlearned models.
Protocol We construct forgetting dataset Dr by randomly sampling a portion of the original |Prl training data Dtr, (i.e., D E {5%, 10%, 20%}),, and use the remaining data D, Dir Dr to train retrained model he From both D, and D., then sample n E (400, 1000, 2000) instances repeatedly for m times to create m subsets from each. These 2m subsets serve as the evaluation targets, with known labels indicating whether each subset originated from the training data of he i.c., {S,, 1) c. } and {(S,, 0) C DrJ. We then apply our method to each target subset S, to classify its in- or out-of-training status with Eq. The classification Fl score over the 2m subsets indicates how well our method can distinguish between in- and out-of-training data under the he
4.1.1 KERNEL BANDWIDTH a
HSIC adopts the Gaussian kernel by default, defined
k(x,, x²) exp (- (-1x-=x' x x'112>
(4)
20
where a is the kernel bandwidth. A smaller a results in more localized kernel that is sensitive to fine-grained differences between samples, which may amplify noise and lead to unstable estimates. In contrast, larger produces smoother kernel, capturing broader structures but leading to higher similarity across all sample pairs.
In this part, we investigate how sensitive our method is to different values of a and aim to identify the effective operating range that yields consistently high performance. We consider two heuristics for selecting a: the square root of the activation dimension, i.e., vdim (Liu al., 2020), and the widely adopted median heuristic (Schölkopf & Smola, 2002; Garreau al., 2017) based on pairwise distances be- tween samples. To carry out this analysis, we use the output of hp and uniformly sample 20 candidate values of from a continuous range between I and max( / dim, Median), and eval- uate the resulting Fl score under each o value.
1.0
SVAINN
0.4
SV-RRekkek1B
0.2
C300 Raa-RRa
Trny Raa--e1131
Figure 3: Effect of kernel bandwidth on Fl score. The solid lines show Fl score trends across different values. Vertical dashed lines indicate the heuristic of /dim, while dot- ted lines correspond to the median heuristic.
5

---

Published as a conference paper at ICLR 2026
Table 1: Fl score of distinguishing infout-of-training status of target S, across the amount of Dr and the size of S.
R
5%
10%
20%
[S]
400
1000
2000
400
1000
2000
400
1000
2000
SV-AIICNN
0.62+0.
0.74+0.08
0.83+0.09
0.50+0.21
0.62+0.06
0.73+0.12
0.7240.03
0.82+0.08
0.90=0.06
CI0-ATICNN
0.69+0.07
0,79+0.05
0.8100.14
0.61#0.05
0.6640.12
0.82+0.09
0,45+0.00
0.71=0.18
0.79:0.09
SV-ResNet18
0.71+0.09
0,9140.01
0.93
0.63a0.10
0.7840.09
0.85+0.14
0,90s0.03
0.9620.02
0.97:0.05
CIO ResNet18
0.8720006
0,9740.02
0.9920.01
0.8840.04
0,95a0.06
0.97=0.03
0.8640.
0.9620.03
1.0000.00
R8
0.99:0.01
100+00.00
1.00=0.00
0.97=0.05
1.00+0.00
1.00=0.00
0.99w00011
1.00+0.00
L00g0.00
Tiny-ResNett8
0.70+0.06
0.78+0.06
0.92=0.05
0.81=0.08
0.92#0.03
0,98=0.02
0.78+0.05
0.9020.06
0,98=0.02
From Figure 3, observe that the choice of kernel bandwidth a has a significant impact on the performance of our method. For most dataset-model combinations, the Fl score improves rapidly with increasing o and reaches a plateau within moderate range. Notably, extremely small values result in unstable or poor performance, likely due to overly localized kernels that are sensitive to noise. Conversely, excessively large values may overly smooth the kernel, resulting in loss of discriminative power and decline in performance. Compared with the median heuristic (dotted lines), the heuristic
din (dashed lines) generally falls within the high-performing regions, supporting its effectiveness as a practical choice. the rest of the experiments, unless otherwise specified, we use a dim as the default choice.
4.1.2 IMPACT OF |S| AND R
We use the activation of the penultimate layer, denoted hz(2), and evaluate models at the check- point corresponding to 80% of the total training epochs. We experiment with forgetting ratios R E (5%, 10%, )0% and target subset sizes [S] (400, 1000, 20003.
Table 1 reports the F1 score across various datasets, architectures, forgetting ratios R. and test subset sizes |8). We observe the following consistent trends: Larger S improves performanee. Across all settings, the Fl score improves as the target set size increases from 400 to 2000. This is expected, as more samples provide more stable H (S,, h) estimates and reduce variance in distribution compar- isons. 2) Our method remains effective even with small R. While larger forgetting ratios R (e.g., from 5% to 20%) may introduce more changes in the model's output, we observe that our method already achieves competitive performance when R is as small as 5%. This indicates that even sub- tle representation differences introduced by forgetting a small portion of the data can be detected using our method, demonstrating the sensitivity and robustness of our approach. 3) Model and dataset matter. Our method performs particularly well on ResNet-18 architectures (e.g., CIFAR- and CIFAR-100), achieving nearly perfect accuracy when |S| 1000 on CIFAR-100. In contrast, performance lower but still reasonable on AIICNN model.
4.1.3 LAYER-WISE GENERALITY
We use the checkpoint 80% of the total training epochs, set |8| 1000 and 10%. Be- yond the final output h and the penultimate activations hip we further examine intermediate rep- resentations to assess the generality of our method across different model layers. For the AlICNN architecture, we select the output of four convolutional layers, [Conv2, Conv4, Conv6, Conv8J, as intermediate activations. For ResNet-18, we use activations from its four residual blocks, e.g., [Block1, Block2, Block3, Block4]. We use l E (1, 2,3, 4} to index these intermediate layers, or- dered from early (closer to the input) to later layers. We present the dimensionality of activations across layers in Appendix D.
From Figure 4, we observe that our method achieves better performance in deeper layers-such as the penultimate layer h, and the final output --exhibit higher distinguishability. as they encode more task-specific information. Intermediate layer such as h4 also yields strong signals, particularly on datasets like CIFAR-10 10 and CIFAR-100 with the ResNet-18 18 architecture, where the Fl remains above 0.9. This demonstrates that our method is not confined to final-layer outputs but generalizes well across representational levels. As expected, the F1 tends to decrease move to lower-level
6

---

Published as a conference paper at ICLR 2026
Table 2: Fl score of SDE across train-
1.0
SHANCHI S0-RA 18
CiD-tectent Thy Rembeet
ing progress. The performance of retrained models at checkpoints is in Appendix D.
0.8
0.4
ckpt at
10%
20%
40%
80%
0.2
SV-AIIONN
0.49#0.02
0.51=0.13
0.58+0.17
0.62 06
0.0
P/- 1/4 34p 02
CI0-AIICNN
0.4840.15
0.6620.04
0.5420.18
0.66x0.12
SV-ResNet18
0.58+0.12
0.77=0.05
0.79#0.05
0.78=0.09
Figure 4: Fl score on activations of different in- termediate layers. The x-axis labels from left to right (h h1) represent layers from output to in- put. The reason for the missing bar (e.g., the green bar at h2) is further discussed Section 5.
C10 ResNet18
0.6920.00
0.6120.13
0.72a0.07
0.9520.06
C100-ResNer18
0.59+0.06
0.60=0.11
0.88+0.07
1.0000000
Tlay-ResNet18
0.66#0.22
0.8620.08
0.91
0.9240.03
layers closer to the input (e.g., h2 and h1), especially for shallower networks like AIICNN. This because earlier layers tend to capture general or low-level features that are less sensitive to the presence or absence of specific samples in training. In summary, our layer-wise evaluation confirms the versatility of the proposed approach, making suitable for scenarios involving partial model access, transfer learning, layer-specific unlearning interventions.
4.1.4 TRAINING SUFFICIENCY
We investigate whether the training sufficiency of the model affects the effectiveness of our method.
In this experiment, we fix the forgetting ratio 10%, the target set size [S] = 1000, and use the model's penultimate layer activation (z) for evaluation. We evaluate our method using check- points saved at 10%, 20%, 40%, and 80% of the total training epochs, respectively.
From Table 2, the effectiveness of our method improves the model's training progresses. For example, on SVHN with ResNet18, the Fl increases from 0.58 at 10% training to 0.79 at 40%. On CIFAR1O with ResNet18, the improvement is even prominent, e.g., from 0.69 0.95. This trend suggests that as the model undergoes more training, its internal representations better encode the training data, leading stronger inter-sample dependencies that our HSIC-based method can capture. Conversely, when the model is under-trained (e.g.. at 10%), the representations are not sufficiently informative, making harder to distinguish between in in-training and -ut-on-training. We also report the performance of retrained models on their respective tasks at these checkpoints in Appendix Table 7, from where we can observe that signs of overfitting emerge as early as the 20% training stage, even when the overall model performance is still suboptimal. For example, under the CIFAR (00-ReNNet18 setting, the model achieves only 72.56% training accuracy, yet the test accuracy is only 56.08%, revealing early onset of generalization gap. This observation is also consistent with a widely acknowledged finding in the MIA literature (Hu et al., 2022): models that more overfitted to their training data are generally more vulnerable to MIA attacks.
Notably, even at merely 20% of the training process, our method achieves Fl above 0.6 in most cases, indicating early signs of distinguishable dependence. This shows that our method remains applicable even under partially trained models.
4.2 COMPARISON WITH DISTRIBUTION DISTANCE METRICS
We compare the proposed statistical independence- based method against two commonly used distribution-based metrics, Maximum Mean Discrepancy (MMD) and Wasserstein distance, to show the advantage of statistical dependence- trained model and use the CIFAR-10 10 dataset with the ResNet-18 18 architecture. The evaluation task remains the same as described in Section 4.1: determining whether a target subset S; is part of the training data by comparing h(S,) with reference sets h(Srr) and h(Soor). For distribution distance based metrics, S, is classified as in-training if h(S,) is closer to h(Srr) than to h(Soor), while the opposite assignment applies to out-of- ofttiin-ra samples.
methods. We continue to focus on the controlled re-
According to Table 3, while both MMD and Wasserstein distance exhibit improved performance with larger subset sizes, their accuracy varies significantly with different forgetting ratios and is
7

---

Published as a conference paper at ICLR 2026
Table Comparison with distribution- metrics
R
5%
10%
20%
[S]
400
1000
2000
400
1000
2000
400
1000
2000
MMD
0.6340.07
0.65&0.09
0,87=0.12
0.45&0.21
0.70a0. 13
0.87a0.14
0.63+0.03
0.72+0.08
0.89:0.11
Wasserstein
0.70s0.08
0.7740.10
0.94g0.11
0.5240.20
0.89s0.08
0.98:0.03
0,72:0.02
0.87x0.07
0.99a0.02
SDE (OURS)
0.8720006
0.99s0.01
0.88:0.04
0.95#0.06
0.97=0.03
0.86:0.10
0.96:0.03
generally lower than SDE. Notably, SDE consistently achieves higher F1 scores across all settings, even when the subset size is small (e.g.. |S] 400). This suggests that our statistical indepen- dence-based approach is more robust than distance-based metrics, particularly when the size of the subset is small. We believe that this advantage stems from SDE directly measuring inter-sample dependency structures, rather than relying on marginal distributional shifts. Moreover, distribution- based methods tend to suffer from higher variance and sensitivity to sample size, which may further degrade their reliability in subset-level evaluation.
4.3 EVALUATING UNLEARNING METHODS
All experiments in the previous sections used the retrained model controlled object. this sec- tion, we evaluate unlearned models that come from several representative unlearning algorithms. We consider several widely adopted unlearning baselines, including Random-labe (Fan et al., 2024), Unroll (Thudi et al., 2022), Salun (Fan et al., 2024), and Sparsity (Jia et al., 2023).
Given an original model he trained on the full training dataset D.., we simulate sample-wise unlearning by randomly removing a subset of training data with unlearning ratios R E (5%, 10%, 20%}.
Unlearning Task:
Evaluation Protocol: Given target subset S of 1000 samples and reserved reference subsets of the same size, SrT C D, and SooT C Di.. the evaluation task is to determine whether S is in- or out-of-training data. For statistical significance, we sampled S DD for 100 times and report the number of subsets identified as in- or out-of-training In this case, an effective unlearned model should have more S identified as out-of-training and less as in-training, resulting in higher out-of-training rate (OTR). To demonstrate that our method is still effective on unlearned models, we sample balanced 100 subsets from D, and Die as controlled targets and report the F1 score. Noteworthy, the controlled target subsets are not required in practice.
Other Metrics: In addition to the OTR, which is based on our proposed method, we follow prior sasple-wise unlearning works' experiments (Jia et al., 2023; Fan et al., 2024; Shen et al., 2024) and report other commonly used metrics, including accuracies on training sets, as well as membership inference attack success rate (ASR). Specifically, Acc, and Ace; denote the classification task ac- curacy of the unlearned model on the remaining set D. and forgetting set Dr, respectively. These accuracies are used to evaluate the unlearned models' task utility. For the ASR, we also follow ex- isting unlearning works (Fan et al., 2024; Jia al., 2023) and adopt the prediction confidence-bart attack method MIA methods (Song et al., 2019; Yeom et al., 2018). According to prior works, a desirable unlearning method should have all these metries that are close to a retrained model.
Results: Table 4 shows the results on CIsAR1O- with R 10% A complete result with R E (5%, 10% 20%} J is shown in Appendix F. Except for the Sparsity method, our proposed approach consistently achieves high Fl scores (mostly above 0.88), showing effectiveness in cor- rectly distinguishing whether given subset S is in-training or out-of -ftt--inininng for unlearned models. The Retrain model shows very high OTR (87% for and 94% for hn)) indicating that many of the forgetting subsets are indeed identified as out-of-training model. From the ASR results, all methods except Sparsity exhibit similar membership inference re- sistance compared to the Retrain (all around 0.3), making it difficult to judge their unlearning quality solely from ASR. However, the OTR provides a clearer picture: the Random-late method shows strong unlearning effectiveness, with 84% of forgetting subsets no longer recognized in-training.
desirable property for fully unlearned
8

---

Published as a conference paper at ICLR 2026
Table 4: Evaluating unlearned models on CIFARIO-ReNNeL18 with A 10% Acc, and Acer denote unlearned models' training accuracy on the D, and Dr, respectively. ASR refers to the success rate of MIA. For these metrics, the closer to the retrained model's the better. For our method, the higher OTR indicates more effective unlearning.
h
Method
Acc, (%)
Accy (%)
ASR
F1
OTR (%)
F1
OTR (%6
Retriin
98.57:10.08
93.25-1:045
0.30::0.09
0.94::0003
87.00 10.24
0.95:00.05
94.00:1-4.94
Raed
98.80100004
98.63-10.13
0:29:1::02
0.88::0.12
84.00 13.54
0.91:10.09
83.20:110111
Unroll
99.36=20.05
99.21=:0.11
0,30::0.12
0.88:20.04
2.00:2.19
0,90:00.07
4.40=4..33
Sparsity
92.72:10.93
90.56:2:.82
0.42::0.09
0.62::0.15
50.80=222.61
0.59::0.16
13.80:22.19
98.66:10.03
98.53:10.07
0.29:10.02
0.85:10.12
52.40:1:1186
0.86.10.15
51.80:123.05
In contrast, the Unroll method has an extremely low OTR, suggesting that nearly all forgetting sub- sets are still treated as in-training, indicating ineffective unlearning.
DISCUSSION AND LIMITATION
The selection of and kernel function Currently, our method relies on kernel-based HSIC to capture statistical dependencies among samples. The choice of the RBF kernel and the bandwidth parameter directly affects the sensitivity of our metric. As shown in Figure 3 and Figure 6, the choice of kernel bandwidth critically affects the F1 score. While the heuristic reasonably well in classification settings, it fails to achieve the best results diffusion experiments, seen in Figure 6. This indicates that simple heuristics may not generalize to all scenarios. More adaptive strategies for a selection, or the design of alternative kernel functions tailored to high- dimensional samples, may further improve the robustness and sensitivity of our evaluation approach.
dim works
The selection of reference sets Our evaluation approach relies on reference sets, which can either be 1) prepared by an authenticated third-party auditor and required be involved in training or 2) provided by the model owner and archived by the auditor. The choice of reference sets affects the performance of our method. For example, the missing green bar at h2 in Figure 4 occurs because randomly selected reference sets SIT and Soor fail satisfy (Srr, h) (SSor,, h) under the U-test. Designing strategies for constructing optimal reference sets may be an important direction for improving robustness.
Advantages of SDE We view SDE as a step toward practical unlearning evaluation in real deploy- ment scenarios, for several reasons: (1) requires no retrained reference model: (2) it avoids aux- iliary model training and supports flexible cross-layer evaluation; (3) it provides an independence- based perspective rather than distribution-base criteria that can be gamed when directly optimized; and (4) its subset-level focus aligns more closely with existing practical unlearning workflows.
Rethinking privacy-oriented unlearning evaluation Our experimental results reveal critical discrepancy between existing evaluation metrics and our proposed method. For example, in Table 4, existing metrics would suggest that Unroll is effective, as shows results close to the retrained model. However, evaluation clearly indicates that Unroll fails to remove the influence of for- getting data, with most forgetting samples still identified as in-training. This discrepancy suggests that relying solely on existing metrics may lead overestimating the effectiveness of unlearning methods, motivating a rethinking of how unlearning should be rigorously evaluated.
Unlearning vs. General forgetting While our approach effectively evaluates unlearning. it may also capture general forgetting phenomena caused by representation drift or catastrophic forgetting. Distinguishing intentional unlearning from natural model degradation non-trivial and may require incorporating temporal dynamics additional verification signals.
Beyond a binary decision Our current evaluation distinguishes in- traiinnig- by compar- ing target subset against reference set, which does not fully exploit that HSIC is continuous

---

Published as a conference paper at ICLR 2026
dependence measure. A promising direction is to quantify unlearning by comparing the depen- dence the same target subset Star C Dr across different unlearned models. For example, H (S huml) < (S hum the unlearned model hum! exhibits better unlearning than h hum?
CONCLUSION
We propose Split-half Dependence Evaluation (SDE) for evaluating machine unlearning based on statistical dependence among the unlearned model's output representations. By designing a tailored use of the Hilbert-Schmidt Independence Criterion (HSIC), method enables subset-level evalu- ation without the need for retrained models or auxiliary classifiers. Our analysis shows the success of SDE is because of shared influence component that is introduced in training progress. Extensive experiments on classification and diffusion- based generative models demonstrate that our approach reliably identifies the in-training and out-of-training status of small data subsets and provides clear, robust conclusions, even in scenarios where existing evaluations fail to offer decisive evidence.
ACKNOWLEDGMENTS
CHZ and MX were supported by the Australian Research Council Discovery Project DE2301011 MXL is supported by Australian Research Council (ARC) with the grant number DP230101540. FL supported by the ARC with the grant number DE224100089, L.P20110101, DP230101540 and the NSF&CSIRO Responsible Al program with grant number 2303037.
REFERENCES
Lucas Bourtoule, Varun Chandrasckaran, Christopher A. Ploquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In SP, 2021.
Preston Bukaty. The California Consumer Privacy Act (CCPA): An implementation guide. IT Governance Publishing, 2019. ISBN 9781787781320.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In SP, 2015.
Yunjey Choi, Youngjung Uh Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan S. Kankanhalli. Zero-shot machine unlcarning. JEEE Trans. Inf. Forensics Secur., 18:2345-233 2254,, 2023.
Chang Dong, Zechao Sun, Guangdong Bai, Shuying Piao, Weitong Chen, and Wei Emma Zhang. Trojantime: Backdoor attacks on time series classification. arXiv preprint arXiv:2502.00644 2025.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation. In ICLR, 2024.
Bent Fuglede and Flemming Topsøe Jensen-shannon divergence and hilbert space embedding. In ISIT, 2004.
Sanjam Garg, Shafi Goldwasser, and Prashant Nalini Vasudevan. Formalizing data deletion in the context of the right to be forgotten. Anne Canteaut and Yuval Ishai (eds.), EUROCRYPT, 2020.
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv: Statistics Theory, 2017.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmiid norms. In Algorithmic Learning Theory, 2005a.
Arthur Gretton, Ralf Herbrich, Alexander J. Smola, Olivier Bousquet, and Bernhard Schölkopf. Kemel methods for measuring independence. J. Mach. Learn. Res., 2005b.
10

---

Published as a conference paper at ICLR 2026
Arthur Gretton, Kenji Fukumizu, Choon Hui Teo, Le Song, Bernhard Schölkopf, and Alexander J. Smola. A kernel statistical test of independence. In NewrlPS, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Mem- bership inference attacks on machine learning: A survey, ACM Comput. Surv., 2022.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In NeurlPS, 2023.
Peihai Jiang, Xixiang Lyu, Yige Li, and Jing Ma. Backdoor token unlearning: Exposing and de- fending backdoors in pretrained language models. In AAAI, 2025.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurlPS, 2022.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), NeurlPS, 2023.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. USENIX, 2020.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learn- ing deep kernels for non-parametric two-sample tests. In ICMI., 2020.
Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM, 2022.
Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, XiaoFeng Wang, Haixu Tang, Carl A. Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In EuroS&P, 2020.
Henry B Mann and Donald R Whitney. tically larger than the other. The annals of mathematical statistics, 1947.
On a test of whether one of two random variables is stochas-
Neil G. Marchant, Benjamin I. P. Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In AAAI, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, 2011.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. CoRR, hbs/2209.00299, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells III, and Alejan- dro F. Frangi (eds.), MICCAI, Lecture Notes in Computer Science, 2015.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In NDSS, 2019.
11

---

Published as a conference paper at ICLR 2026
Bernhard Schölkopf and Alexander Johannes Smola. Learning with Kernels: support vector chines, regularization. optimization, and beyond. Adaptive computation and machine learning series. MIT Press, 2002. ISBN 9780262194754.
Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Chen, and Miao Xu. Label- agnostic forgetting: A supervision-fre unlearning deep models. In ICLR, 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. SP, 2017.
Le Song, Alexander J. Smola, Arthur Gretton, Karsten M. Borgwardt, and Justin Bedo. Supervised feature selection via dependence estimation. In Zoubin Ghahramani (ed.), ICML, 2007.
Liwei Song, Reza Shokri, and Prateck Mittal. Privacy risks of securing machine learning models against adversarial examples. In ACM SIGSAC CCS, 2019.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015,
Ayush K. Tarun, Vikram S. Chundawat, Murari Mandal, and Mohan S. Kankanhalli. Fast yet effec- tive machine unlearning. IEEE Trans. Neural Networks Learn. Syst., 2024.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling SGD; under- standing factors influencing machine unlearning. In EuroS&P. 2022.
Yiwen Tu, Pingbang Hu, and Jiaqi W Ma. A reliable cryptographic framework for empirical machine unlearning evaluation. In NeurlPS, 2025.
Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Trans. Emerg. Top. Comput. Intell., 2024.
Dingqi Yang, Daqing Zhang, and Bingging Qu. Participatory cultural mapping based on collective behavior data in location-basee social networks. ACM Truns. Intell. Syst. Technol., 2016.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. IEEE CSF, 2018
Chenhao Zhang, Weitong Chen, Wei Emma Zhang, and Miao Xu. Countering relearning with perception revising unlearning. In ACML, 2024.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used LLMs solely for polishing the writing of the paper, including grammar correction and phrasing alternatives for clarity and brevity. All content generated with the LLM was treated as suggestions and was reviewed, edited, and verified by the authors, who take full responsibility for the manuscript.
The LLM was NOT used in generating technical content; all technical content is author-generated and author-validated
A INSIGHTFUL ANALYSIS
In this section, we provide insight into why the proposed Split-half Dependence Evaluation prov- ably distinguishes subsets drawn from the training data (in-training, IT) from those never used in training (out-of-trainnn, OOT). Intuitively, training progress leaves shared "influence footprint" of each example in the learned parameters; this common component appears in both halves an IT subset. couples their representations, and yields strictly positive HSIC. In contrast, OOT subsets do not affect the training trajectory and contribute no shared influence; consequently. their two halves remain independent.
12

---

Published as a conference paper at ICLR 2026
A.1 PRELIMINARIES
HSIC operator view. For random variables X, Y with RKHS features sP; v HSIC(X,Y) = [[xx Cxv = (X)( px) @ (+(Y) py)].
Here || ||s denotes the Hilbert-Schmidt norm: for an operator A between Hilbert spaces,
A = Σ I|
where {es) is any orthonormal basis of the input space. Equivalently, if A admits singular value decomposition with singular values {1z), then || = Σ, oz
Local linearization of representations.
Around a reference parameter A,, (z; 0) = hz 0.) + + (x),, Jx(x) = Vohr(z;0)lo.-
(5)
with [R((x)] o(]
Split-half cross-covariance.
Given a target subset S with a random split into S₁ and S2, we define H(S, he) = HSTC) he(S2))
With characteristic kernels, HSIC can be expressed through the RKHS cross-covariance between the halves. Using the linearization Eq. 5, the dominant term is
Cs. Lo| (J: - (J((X)A))
& (JI(X')OO-EE) -((())))-EE((()A00) EE)(X'A0))
(6)
where S1 and X' -S are independent draws.
Influence decomposition of parameter shift. Under ERM with (mini-batch) SGD and mild reg- ularity (Koh & Liang, 2017), the trained parameter admits the approximation
1
AA
Σ
I(z),
I(x)
- Vol
TI
JEDD
where H is damped Hessian PSD curvature proxy. This holds first-order approximation via influence functions.
A.2 CASE 1: IN-TRAINING SUBSET (S C Dt))
In this case, the influence decomposition Eq. can be refined
1 I(x)+
Σ I(x).
A0
(8)
IS
A0g
ABreat
Here Ads the shared component contributed by S itself. Note that both halves S₁ and S₂ inherit this same Ads, which creates correlation across the split.
Starting from the split-half cross-covariance Eq. 6 and the IT decomposition Eq. write
A0g + Alreat-
Abbreviate = Je(X) and A' = Je(X'). and their centered versions Ā = A- E.A, A' A' - EEA' Then Eq. 6 is
Cs..s E (AAP EAAØ) @ (A'A0 (A'A8-EA'A9) EAADA]]
(AS00 (A'
=I((A(A0s + A0 @ (A'( ADrest))
T Tst T.s + T.r.
13

---

Published as a conference paper at ICLR 2026
where the four blocks are
Tss E((A0(s) @ (A'A s E((AS0rest @ (A' A8)).
Ts, E[(AA0s) @ (A'
T.r = [[(A00 prsst) ) @ (Arstr
Condition on the fixed split so that Ads and Afrest are constant vectors, while X S1 and X' S are independent. Then by independence,
(X) & v(X)) ((U( & B[V()))
Hence
Ts- [[ @ E[A'A0 rest = (B[A) X (B[AA)A0rest) = O,
and symmetrically T O. Moreover,
Trr = E[AA0 rose @ E A 0,
again because E[A] = and E[A'] = by centering. Therefore,
Cs Tss (AA0s) @ (A'A0s)).
(10)
Write A0g via the halves:
A0s = (Is, + Is.),
Is.
E I(x), Is. = Σ I(z),
#ES1
ZES2
which are constant vectors given the split. Then
AAOs (Als. Alss): and Eq. 10 becomes
A'A0s 1 (A'ls, + AA'lss),
n1
(AAA0)) (A'AAs)
1
B[(Tss + ÄIs,) @ (A'Is, A1sss)
n
1
Σ
Σ
E (AL.) & (A'b)).
(11)
n²
(
(S)
Since Ia, To are constant and we already work with the centered covariance (Ā = A- - EA, Ā' =
A' EA'), we can have
(J(X)AAs) & (J(('))Ass]
Σ E E((J(()))) & (J)(' (12)
n
aE(S SS)
where Is, = E.es, I(x) and = I(x) are constant vectors given the split.
Eq. 11 contains four terms: two "same-hali" terms (a = b) and two "cross-half" terms (a # b).
Conditioned on fixed split, X S₁ and X' 82 are independent, hence
(X) 8 J((X')] [X)] @ E[(e(X')).
Therefore, after centering, each same-half term cancels:
E((J{(X)ss)) @ - (
E [I (J. 11
-[J((x)sss 8 (()(('))s))
Hence only the cross-half contributions remain:
B[(J{(X)ss, & (Je(X') + (X()())(s,, @ (J(('')ss))) These cross cross-terms are non-vanishing because the shared component
(13)
A&g ( + Is.) # enters both halves (under non-degenerate Jr statistics). Consequently, and H(Sm, he) || l1
cT S,
14

---

Published as a conference paper at ICLR 2026
Intuition (IT case). Even though S1 and S2 are disjoint subsets, they not independent once has influenced the parameters: both halves share the same "fingerprint" Adg in the model. This induces dependence across the split and guarantees a strictly positive HSIC.
A.3 CASE 2: OUT-OF-TRAINING SUBSET (S n Die 0)
When S' is completely unseen during training, does not contribute to A0. Formally,
A0s 0,
A A0rett
Substituting into Eq. 6, we obtain
COOT = B (J.()))0sss (J) 8 (Ja(X))A (EJ((X')
(14)
By comparing with the Eq. 13 in the IT case, there is no shared AOs and cross-half influence in the OOT case. Factorizing the constant vector 20rest this becomes
[E(J(()) @ J((X) - E[)((X) & [[)(()))) (ABrest & A.rest).
Since X S1 and X' S2 independent draws from disjoint halves of an OOT set,
(X) Je(')) E[J((X)) B[))
(15)
and the entire expression vanishes:
Sy 0,
(SooT, hε) 0.
Intuition (OOT case). OOT subsets never influenced the learned parameters, so they do not inject any shared component across the halves. After conditioning on the split, Sj and S2 are independent samples with no common "training footprint".
10a
0.030
D-value S h) > HIS H(Scross crossy h))
0.025
10
H(S, h) Values
0.020
10-22 22
Sgame
10
p-value
10-38
6
8
10
88
10-54
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85 90 95 100
Training Epoch
Figure 5: (S, h) distribution changes along the training epoch. The dashed gray line is the p 0.01 baseline.
A.4 FIXED BATCH MEMBERSHIP EXPERIMENT
In this subsection, we use toy experiment to better reflect our main intuition, which is also the basic idea presented in the above analysis dependence among outputs arises because training samples co-occur in gradient computations and parameter updates during training.
The toy experiment is based on the following insight: if the model is trained with fixed (rather than randomly sampled) mini-batches, then samples within the same batch should exhibit sig- nificantly stronger dependence compared to samples drawn from different batches. To validate
15

---

Published as a conference paper at ICLR 2026
this, constructed binary classification task with 10,000 data points and 10-dimensional features. We set the batch size to 64 and fixed the batch membership at the beginning of training. A single hidden layer MLP, hidden dimension 128, is used the model. Under this setup, we expect the dependence within a same-batch subset Saame to be significantly larger than that of a cross-hatch subset Seuss that is, (Ssame, h) (SScruss)) ht).
In Figure 5, the gap between the (Samee: h) and H (S h) goes larger along with the training steps. The result aligns with this expectation, with a statistical test yielding p 2.11 x 10-54 0.01, supporting the conclusion that H (Ssame,))) > (Sersss) h).
B RELATED WORKS
B.I UNLEARNING EVALUATION
Machine unlearning (Bourtoule et al., 2021; Nguyen et al., 2022; Xu et al., 2024) aims to remove the influence of specific training data from machine learning models. enhances user privacy by complying with regulations such as the "right to be forgotten" (Garg et al., 2020; Bukaty. 2019) and can also mitigate the impact of errors or adversarial contamination in training datasets (Cao & Yang, 2015; Marchant et al., 2022). The effectiveness of unlearning algorithm depends its goal and scenario (Kurmanji et al., 2023). When the goal is to remove erroneous or adversarial knowledge (Dong al., 2025; Jiang et al., 2025; Liu et al., 2022), unlearning effectiveness is often measured by the drop in adversarial attack success rate while maintaining overall model utility. In this paper, focus on the privacy-oriented unlearning evaluation.
A common evaluation strategy for privacy-oriented unlearning is to use a model retrained from scratch without the forgetting data as the gold standard. Existing approaches assess how similar the unlearned model is to this retrained model in terms of output distributions (Nguyen et al., 2022) or relearning time (Tarun et al., 2024; Zhang et al., 2024). Another intuitive approach leverages Membership Inference Attacks (MIA) to determine whether the forgetting data still leaves de- tectable trace in the model. Most prior works still compare the MIA success rate of the unlearned model against that of a retrained model. In contrast, our method enables post-unlearning evalua- tion without requiring a retrained model. It directly measures whether the influence of the forgotten data persists in the model through statistical dependence analysis, offering a scalable and practical mrivacy-orented evaluation approach.
A recent unlearning evaluation study (Tu et al., 2025) also explored the idea of "split sets." In (Tu et al., 2025), their methodological principle is the cryptographic indistinguishability that requires the unlearned model be computationally indistinguishable from retrained models across adversar- ially chosen dataset partitions. To test this, they split the dataset multiple times to create different training scenarios for indistinguishability evaluation. In contrast, our methodological principle is statistical independence, which directly tests whether training-induced dependencies persist in the model representations of a given subset. Specifically, we split the given subset into two halves and use the HSIC test to measure the dependence between them. (Tu et al., 2025) requires training mul- tiple models across dataset splits to construct distinguishing games. Ours only conducts statistical dependence testing on the target model's outputs, and no auxiliary model training is needed.
B.2 MEMBERSHIP INFERENCE ATTACK (MIA)
Membership inference attacks (MIAs) (Hu et al., 2022) aim to determine whether a specific sample was included in a model's training data. MIAs have been widely used as a privacy auditing tool and have strong conceptual alignment with unlcarning evaluation, since successful unlearning should erase any membership signal of the forgetting data. A branch of existing MIA techniques (Shokri et al., 2017; Leino & Fredrikson, 2020; Long et al., 2020) trains a binary classifier to identify data sample's membership regarding the target model's behavior. Another branch of the MIA exploit model confidence (Salem et al., 2019) or loss values (Yang et al., 2016) to identify the membership of a data sample. Existing MIA methods often require additional model training (e.g., shadow models or binary attackers) or access data labels compute losses, gradients, or prediction correctness. While these approaches can determine the membership of individual samples, they incur significant overhead and rely on extra assumptions.
16

---

Published as a conference paper at ICLR 2026
In contrast, method leverages key property of most unlearning scenarios: forgetting typically targets subset of the training data rather than isolated samples. By analyzing the statistical depen- dence within group of samples, our approach provides simpler and reliable evaluation. It requires no additional model training, no access to the original training procedure or hyperparameters, and no labeled data, making it practical tool for post-unlearning privacy assessment.
B.3 STATISTICALLY SIGNIFICANT DEPENDENCE
Measuring statistical dependence between random variables is fundamental problem in both statis- ties (Gretton al., 2005b) and machine learning (Song et al., 2007). Beyond classical measures such Pearson correlation and Mutual Information (MI), the Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005a; 2007) offers more general framework, capable of detecting arbitrary dependencies in high-dimensional spaces without requiring explicit density estimation. Our work leverages HSIC to quantify the residual dependence among the representations of a set of samples and build pipeline for evaluating unlearning effectiveness. If unlearning is effective, the repre- sentations of the forgetting data should appear statistically independent, indicating the removal of training influence. By focusing group-level dependence rather than individual sample behavior, our approach offers a robust and statistically meaningful criterion for post-unlearning evaluation.
C ALGORITHM FOR UNLEARNING EVALUATION
We decompose the overall unlearning evaluation process into three:
Algorithm (estimate_hsir distribution) first estimates the distribution HSIC values for target subset by repeatedly permuting the data. This forms the statistical foundation for detecting whether subset shows significant dependence.
Algorithm 1 estimate saaitte_hhic_iiiie hisccdiistibuttion)) # Section 3.1
Require:
Model h
Subset S
Permutation times T
Ensure: [S, h)
S [: (Sl(ennS////) /2)
S S [Len(S) (S) [2
H = []
for in rT)ge(T) do
S RandomShuffle(S.)
V (SII
H. H.append(V))
end for return H
Algorithm (is in training) then evaluates whether a candidate subset belongs the training set. It compares the HSIC distribution of the target set with reference in-training and out-of-training sets using the closer to the in-training distribution.
Shannon divergence (JSD), classifying the set as in tn-training if its HSIC profile is
Finally, Algorithm 3 (unlearn.ccm (unlearn_eval) quantifies the OOT rate of the unleamnd model by sampling multiple subsets from the forgetting data D;, and checking how many of these subsets are success- fully identified as out-of-training A higher OOT rate indicates more effective unlearning, more forgetting data are recognized as being removed from the training set.
D IMPLEMENTATION DETAILS
For the experiments on the classification task, we used mini-bater stochastic gradient descent (SGD) with weight decay of 10-4 The batch size 256 for SVHN and CIFAR-10, and 128
17

---

Published as a conference paper at ICLR 2026
Algorithm 2 is.iinnnne trainnnn) # Section 3.2
Require:
Model h
Permutation times T
Target set Star
Reference sets SIT and SooT
Ensure: Is Star in-training data?
(S h) = estimate_iiiee _aaiimtttehhiiheeeeee distribution(h,, (Stre),
H (Sr, h)) estimate www.hhhchhccccccc fisiiddittribution(h H (S
(SoOT,
siiimatt_hsic_disieee (H (SooT))
DSStar Srr, h) JSD(H (Strr Sar. h), ((Sr.h)) h))
DSSia SooT. h) = J)S(H (S h), H (Soor. h))
return (Star; SIT, h)) < (SSrr SooT, h)
Algorithm 3 unlearn eval()
Require:
Unleamed model mm
Permutation times T
Forgetting training Dr Reference sets Syr and SooT
Size of target
Number of target sets m
Ensure: OOT
larget list
for in range(m) do
sampling S; Dj, |S =
target.list.appendde
end for OOT-Count-1
for Star in target list do
if not is in training(h "TS.a S.SSa.rS then
OOT CoT_CCunt+= end if end for
return OOT Count / m
for CIFAR-100 and Tiny-ImageNet. The initial learning rate was 0.01 for SVHN and 0. for all other datasets. models were trained for the number of epochs listed Table 5, and the corresponding training. evaluation, and test accuracies are also reported in the table.
Table 5: Training, eval, and test accuracies for different architectures and datasets.
Table 6: Activation dimensionalities of different layers.
Dataset-Arch
TRAIN
EVAL Ace
TEST Age
h1
h,
h
SV-AIICNN
100.00:0.00
94.90-10.05
SVIN- AIICNN
12288
10
CIO-ATICNN
50
99.41 +0.02
91.9511.244
91.63-1-0099
C1O. AIICNN
98304
49152
12288
12258
192
10
SV-ResNer18
100.0000000
94.6411.17
94.85+0.10
SVHN-ResNe188
32768
8192
512
C10-RosNet18
100
99,94 =0.01
93.48:0.35
93.48-+0.2
C10-ResNet18
65536
32768
16384
8192
512
10
100
99.92 10.01
72.75:1.338
73.474:0.1
C100-ResNett8
65536
32768
16384
8192
512
100
Tiny ResNe118
100
91.10 10.23
57.20.10.36
57.75.10.50
Tiny ResNet18
262144
131072
65536
32768
512
200
The Figure 4 in the main paper shows the effectiveness of our method across model layers. Table presents activation dimensionalities of different layers. For each layer activation, set the /dim by default. Since the dimensionality of the last layer's activation (the logits) of the 10-class tasks is too small, i.e., Vii < 3, manually set the 128 for them.
18

---

Published as a conference paper at ICLR 2026
In the Table of the main paper, explore the impact of training progress on the effectiveness of our method. To demonstrate the models' training sufficiency, we report the model's performance (TrainAcc/TestAcc)) at various checkpoints during training in Table 7. We can infer from the table that models exhibit almost no overfitting before 20% of training, but they are not yet fully trained to achieve high performance.
Table 7: Task utilities of models across training progress.
Data-Arch
Acc
10%
20%
40%
80%
SV-AIICNN
Train 88.09
95.73
98.51
99.92
Test
88.16
92.97
92.54
94.26
Train 92.77
99.21
100.00
100.00
CIO-ATICNN
Test 90.49
92.07
94.76
94.83
SV-ReaNet18
Train 77.52
84.92
89.22
98.65
71.42
78.12
91.49
C10-ResNet18
Train 76,56
87.20
91.50
99.85
Tost 71,90
80.86
85.16
93.33
Train 58.68
72.56
81,99
99.88 88
C100-ResNe188
Test
52.65
56.08
57,14
73.31
Train
60,30
84.10
91.04
91.11
Teny-ReaNet88
Test 53,94
$7.92
57.75
$7.69
All the experiments are conducted on one server with NVIDIA RTX A5000 GPU (24GB GDDR6 Memory) and 12th Gen Intel Core i7-12700K CPUs (12 cores and 128GB Memory). The code was implemented in Python 3.12 and CUDA 12.4. The main Python packages' versions are the following: Numpy 2.2.5; Pandas 2.2.3; Pytorch 2.7.0; Torchvision 0.22.0.
E CASE STUDY: DIFFUSION MODELS
We further apply our method to generative models, specifically focusing on Elucidated Diffusion Models (EDM) (Karras et al., 2022), one of the most advanced diffusion--assed generation frame- works. Experiments are conducted datasets (Karras et al., 2019). For each dataset, we randomly sample 50% of the data to form the forgetting set Dr, and follow the official EDM training configuration' to train retrained model on the corresponding remaining set D,. The EDM architecture adopts a U-Net (Ronneberger et al., 2015) structure consisting of an encoder and decoder. For our evaluation, output as the input of HSIC, which yields a flattened dimension of dim = 16384. In computing HSIC, we set the kernel bandwidth values span the range from the full dimensionality.
the CIFARIO, AFHQv2 (Choi et al., 2020), and FFHQ
extract the encoder's
CITARLO
AFHOV2
0.6
2500
5000
7500
140000
12500
15000
Figure 6: Fl score across on the diffusion model.
From Figure 6, the Fl score rises sharply a increases from I to roughly 100, after which stabi- lizes and maintains high performance across a broad range of a. AFHQv2 and FFHQ consistently achieve high Fl scores (above 0.9). contrast, CIFARIO exhibits gradual decline as a continues to increase, mirroring the observation in classification tasks. This is likely due to its lower resolution, which limits the representational richness of the encoder features. The zoomed-in inset highlights the sensitivity in the small-o region, where excessively narrow kernel bandwidths fail to capture meaningful dependencies in the high-dimensional encoder features. Beyond a ze 100, the results
httpp:://gi thub. com NV1 waw.aaa//ee
19

---

Published as a conference paper at ICLR 2026
stabilize, suggesting that moderate bandwidths are sufficient for reliable HSIC estimation. Follow- ing the vdim heuristic (a 128), the method achieves Fl score of approximately 0.8 across all datasets, confirming the heuristic as a practical choice for real-world high-dimensional generative models, with further tuning providing additional robustness.
F MORE RESULTS ON EVALUATING UNLEARNING METHODS
In the section, we use the CIFARIO-ResNedt setting as an example and conducted experiments with (5%, 10%, 20%). We provide complete results in Table 8.
Table 8: Unlearned models' results on CIFARIO-ResNet18 with E (5%, 10%, 20%}. Acc, and Ace; denote unlearned models' training accuracy the D, and Dr, respectively. ASR refers to the success rate of membership inference attacks. For these metrics, the closer to the retrained model's the better. For our method, the higher OTR means that more forgetting data has been identified as out-of-training data, indicating more effective unlearning.
By
Method
Acc, (%)
Aory (%)
FI
OTR (%) f
F1
OTR (%)
Retruin
98.70.10.03
93.66.10.10
0.32.10.12
0.85
90.60.1 10.84
0.92
93.20.1..57
5%
RandLabel
98.86=20.03
98.64:10.13
0.25:1:.03
0.89::0.99
29.00=26.26
0,90::0.09
76.40:128.08
Unroll
0.003
99.24.10.22
0,28.10.10
0.88
0.80
0.88 1.009
1.00.12.00
Sparsity
93.75
91.524:0.38
0.44:10.88
0.66:1::29
31.00-31.12
0.66-10.31
62.20:1:33.07
SalUn
98.73=0.002
98.54:b0.14
0.23
0.84:0.11
38 18.80=21.33
0,87#0.11
37.20=226.61
Retmin
98.57=0.08
93.25::0.45
0.30:1:.09
0.94
87.00 10.24
0,95::0.05
94.00-1-4.94
10%
RandLabel
98.80110.04
98.63.10.13
0.29:1:002
0.88.10.12
84.00.113.55
0.009
83,20 10.11
Unroll
99.36:10.05
99.211:.11
0.30:10.12
0.88:10004
3.00:2.19
0.90
4.40:::4.01
Sparsity
92.72=0..3
90.56++082
0.42::0.9
0.62+:0.15
50 80=02.261 61
0.
53 80=224.19
SalUn
98.66:20.03
98.53::0.07
0,29::0.02
0.85:::0112
$2.40:1:1.86
0.86:20.15
51.00-223.05
Retrain
98.58=00044
92.93=00.27
0.25+00.7
0.97+00.22
99. 99.60=:0.49
0
99 80-+0.00
20%
Randl ahel
98.65:10.06
98.64:10.05
0.46:10.02
0.96:10003
72.60-:20.5
0.96-10.03
23.20
Unroll
99.41:20.04
99.27::0.06
0.24:0.112
0.84::.20
24.40=27799 95
0
7.00 72
Sparsity
94.28:20.58
92.00:10.66
0.35:20.07
0.75
62.60=18.11
0.73+00.9
47.40=1:3.02
SalUn
98.51:10.07
98.54:1.0008
0.48:10.03
0.93.10.00
47.80:125.14
0.93:10005
39.60:127.41
Across all forgetting ratios, our method consistently achieves high Fl scores (mostly above 0.85),
indicating a strong ability correctly distinguish whether a given subset in-training or out-of- training. Notably, the Retrain baseline shows very high OTR (over 87% for h and 93% for hp at R 5%), confirming that most forgetting subsets are successfully recognized as out-of-training an ideal unlearning behavior.
From the ASR perspective, all methods except Sparsity achieve membership inference resistance comparable to the retrained model (approximately 0.25-0.32). In this regard, Unroll appears to achieve the most effective unlearning, as its ASR is closest to that of the Retrain model. However, the OTR metric provides clearer insights:
RandLabel demonstrates strong unlearning effectiveness with high OTR (79%, 84%, 72% for R = 5%, 10%, 20%), indicating that large portion of the forgetting subsets are no longer recognized as in-training.
Unroll consistently yields extremely low OTR (below 5%), suggesting that most forgetting subsets remain in- in-training, revealing ineffective unlearning despite its high accuracy.
Sparsity achieves moderate OTR (2%), but suffers from low accuracies and higher ASR, showing unstable unlearning quality.
SalUn achieves intermediate OTR performance (38%-52%). indicating partial unlearning but not as effective RandLabel or Retrain.
Overall, combining ASR with OTR reveals that RandLabel and Retrain exhibit the most desirable unlearning behavior, while Unroll fails effectively forget the target subsets.
20

---

Published as a conference paper at ICLR 2026
G COMPUTATIONAL COST ANALYSIS
Our proposed SDE consists of the following main computational steps:
Network inference. Both the test sample size and the network architecture influence this step's s runtime. We exclude this cost from analysis for two reasons: (I) this is common step for almost evaluation methods, including MIAs and ours; (2) practice, inference only needs be performed once, and the network outputs be reused by subsequent steps across different evaluation methods.
2. HSIC calculation. This step requires O[ d) matrix operations, where representa- tion dimension. It can be parallelized efficiently GPUs. A promising direction for future optimization is to incorporate Nyström kereel approximation to reduce the effective kernel matrix size.
3. Repeatively sample Star DD; for m times for counting OTR. This results m x above cost.
The overall time complexity could be approximated as O(m x ||| x d).
To demonstrate how m. d, and [S] influence runtime in practice, conducted brief experiment under the CIFARIO-ReSNetl setting. The table below reports wall-clock time in seconds. The overall runtime for each entry should include the network inference time corresponding its subset size. As expected, the runtime increases with the number of repetitions m, representation dimension d, and subset size SS|:
Table 9: Wall-clock time cost (seconds) with different m and |S] when 512.
Inference Time
50
100
200
S 400
0.29+
9.51
18.76
37.76
S 1000
0.72+
11.06
21.92
45.85
S 2000
1,43+
29.02
57.70
116.04
Table 10: Wall-clock time cost (seconds) with different m and |S] when 8192.
Inference Time
m = 50
m
100
m 200
S 400
0.29+
13.46
26.55
52.35
S 1000
0.72+
27.15
54.18
107.99
S 2000
1,43+
84.84
168.30
335.67
21


Reviews:
[{"review_id": "hWo3FvHPjC", "reviewer": "Reviewer_kGt2", "summary": "This paper proposes a new metric for evaluating unlearning methods, leveraging a clever connection to statistical independence. The main idea behind the paper is to measure the statistical dependence (using a method called the Hilbert-Schmidt Independence Criterion) between different subsets of examples, and use the computed (in)dependence scores as a proxy to evaluate unlearning efficacy. This allows one to evaluate unlearning without needing ground-truth retrained models.", "questions": "- Is there a simple theoretical setting where this evaluation is exactly the right thing to do?\n- Did the authors try other kernel functions outside of RBF?\n- Can the authors provide a computational cost analysis of their method? What are the main computational steps, and how does the cost scale with the various dataset sizes?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies an interesting and important problem (improving the efficiency of unlearning evaluations)\n- The idea to use statistical dependence testing as a proxy for retraining is quite interesting and novel. It also seems quite convenient to implement, and thus practical to run in actual unlearning setups.\n- The paper conducts a thorough ablation study to understand the effect of different design choices on the efficacy of their approach", "weaknesses": "- The theoretical analysis seems quite handwavy and makes lots of approximations without really justifying why we might expect them to be true. To me this is the main concern with the paper, and one that permeates throughout the rest of the weaknesses (see also Q1).\n- In Table 4, it's seems concerning that the retrained model does not get 100% according to the metric - doesn't this suggest that the metric is either overly sensitive or otherwise misspecified with respect to the unlearning objective?\n- The fact that the method hinges on the selection of good reference sets also limits the practical applicability of the algorithm (this limitation is explicitly acknowledged by the authors).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4o2yoBFhmm", "reviewer": "Reviewer_UFX4", "summary": "This paper introduces *Split-half Dependence Evaluation (SDE)*, a new way to measure how well a model has “forgotten” data during machine unlearning. Instead of retraining models or using membership attacks, it checks the statistical independence of model outputs on data subsets using the Hilbert–Schmidt Independence Criterion (HSIC).", "questions": "1. Missing reference: Recently, there have been many related works focusing on unlearning evaluation, e.g., [1], where the idea of split sets is also explored.\n2. Weakness 1: Can you explain or provide additional experiments that demonstrate some desirable properties for the proposed evaluation metric, compared to others? Also, can you explain why your evaluation metric is better compared to the existing literature?\n3. Weakness 2: Can you provide some brief justification for each of the mentioned components, besides your main conceptual novel proposal (SDE)?\n\n[1]: Tu, Yiwen, Pingbang Hu, and Jiaqi Ma. Towards Reliable Empirical Machine Unlearning Evaluation: A Cryptographic Game Perspective.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear presentation and good illustration.\n2. Conceptually new evaluation framework.", "weaknesses": "1. Unclear motivation: The benefits of the proposed method should be better explained and carefully compared with other proposed methods. For a proposal of an \"evaluation\" method, it is especially important to measure/prove/validate its properties, such as robustness, consistency, among others.\n2. Unjustified design choice: Some of the design choices are not well-explained, e.g., HSIC with specific RBF kernels, Mann-Whitney U-test, Jensen-Shannon Divergence, among others. To me, it feels artificial to see these components being introduced to the method without a clear justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLMxrhWvs8", "reviewer": "Reviewer_3Ap9", "summary": "This paper examines machine unlearning evaluation at the subset level. Motivated by insufficiencies in existing evaluation methods such as retraining (which is computationally expensive) and membership inference attacks (MIA) (which lack effectiveness), the authors propose a tailored metric based on the Hilbert-Schmidt Independence Criterion (HSIC) to measure statistical dependence between model representations and the unlearned subset with a proxy. Experiments demonstrate that the proposed method serves as a reliable evaluation tool for machine unlearning.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. I find the paper very well written and pleasant to read. It is sufficiently motivated by an important problem in machine unlearning and proposes an elegant solution. The presentation is clear and easy to follow.\n\n2. The adoption of subset-level statistical dependence as an evaluation metric is both interesting and clever. It provides a well-defined tool for assessing model-data interactions in machine unlearning (and potentially for measuring neural network memorization more broadly). I believe the proposed method has strong potential to become a standard evaluation tool in the unlearning literature.\n\n3. The paper potentially opens many promising future research directions. I would be particularly interested to see its adaptation to other foundation models, such as those based on contrastive learning methods and generative models.", "weaknesses": "1. It appears that the size of the unlearning subset plays an important role in the effectiveness of the proposed method. The experiments demonstrate that SDE works well for unlearning sets comprising 5-20% of the training data, which represents a relatively large proportion. However, consider a scenario where a user requests the unlearning of only a few samples: would SDE remain effective in this case? In other words, is it possible to quantify the minimum subset size at which SDE provides a reliable evaluation?\n\n2. The experiments show that Unroll obtains very low OTR. Is there any explanation for this phenomenon? Moreover, I am curious whether it is possible to attribute unlearning effectiveness at the individual sample level within the subset. While this may seem overly ambitious from a statistical perspective, I wonder if the authors have any insights or preliminary thoughts on this direction.\n\nNote that neither question necessarily requires additional experiments. The authors are welcome to include a discussion of these points and propose reasonable directions for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WFRBxQZ7DV", "reviewer": "Reviewer_QQ14", "summary": "This paper proposes to use Hilbert–Schmidt Independence Criterion to evaluate the effectiveness of unlearning methods. This novel, statistic-based method facilitates evaluation without a retrained reference model or shadow models.", "questions": "The authors claim that HSIC evaluation does not require a retrained reference model. But I'm wondering: Is it possible that one day a new unlearning method achieves higher OTR than a retrained model's? If so, how do we analyze this phenomenon?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed method is well-motivated, novel, clear, and effective. This paper is well-presented and easy to read.\n2. The research questions studied in 4.1 are important.\n3. The discussion in Section 5 is comprehensive and valuable.", "weaknesses": "1. Previously, unlearning methods usually played with the metrics, including accuracies and MIA results, via cherry picking hyperparameters with the best performance. It seems HSIC won't stop this game, but just provides one more metric to fit.\n2. I think HSIC might be extended for unlearning in generative models, but the authors did not discuss it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)