Knowledge Distillation as Decontamination? Revisiting the “Data Laundering” Concern in Classification Tasks
Authors: Hengyu Luo
Raúl Vázquez
Timothee Mickus
Filip Ginter
Jörg Tiedemann
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
Data Contamination
Benchmark Integrity
Data Decontamination
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Concerns have been raised that knowledge distillation may transfer test-set knowledge from a contaminated teacher to a clean student---a "data laundering" effect that potentially threatens evaluation integrity. In this paper, we assess the severity of this phenomenon. If these concerns regarding data laundering are minor, then distillation could be used to mitigate risks of direct data exposure. Across eight classification benchmarks, we find that substantial laundering is the exception rather than the rule: unlike the large performance gains from direct contamination, any accuracy inflation from laundering is consistently smaller and statistically insignificant in all but two cases. More broadly, using sample-level analysis, we find that the two phenomena are weakly correlated, suggesting that laundering is not simply a diluted form of contamination but a distinct effect that arises primarily when benchmarks exhibit large train–test distribution gaps. Motivated by this, we conduct controlled experiments that systematically enlarge the train–test distance on two benchmarks where laundering was initially negligible, and observe that laundering becomes more significant as the gap widens. Taken together, our results indicate that knowledge distillation, despite rare benchmark-specific residues, can be expected to function as an effective decontamination technique that largely mitigates test-data leakage.

Published as a conference paper at ICLR 2026
KNOWLEDGE DISTILLATION AS DECONTAMINATION? REVISITING THE "DATA LAUNDERING" CONCERN IN CLASSIFICATION TASKS
Hengyu Luo', Raúl Vázquez', Timothee Mickus', Filip Ginter Jörg Tiedemann 1.3 l University of Helsinki, University of Turku, $ELLIS Institute Finland
firrstnae,, Lastname@hellinkiiff 2figint@utu.fl
ABSTRACT
Concerns have been raised that knowledge distillation may transfer test-set knowl- edge from contaminated teacher clean student-a "data laundering" ef- fect that potentially threatens evaluation integrity. In this paper, we assess the severity of this phenomenon. If these concerns regarding data laundering are minor, then distillation could be used to mitigate risks of direct data exposure. Across eight classification benchmarks, we find that substantial laundering is the exception rather than the rule: unlike the large performance gains from direct contamination, any aceuracy inflation from laundering is consistently smaller and statistically insignificant in all but two cases. More broadly, using sample-level analysis, we find that the two phenomena are weakly correlated, suggesting that laundering is not simply a diluted form of contamination but a distinct effeet that arises primarily when benchmarks exhibit large train-tes test distribution gaps. Moti- vated by this, conduct controlled experiments that systematically enlarge the train-test distance on two benchmarks where laundering was initially negligible, and observe that laundering becomes more significant as the gap widens. Taken together, our results indicate that knowledge distillation, despite rare benchmark- specific residues, can be expected to function as an effective decontamination technique that largely mitigates test-data leakage. Code is available at https: //github. http://nooo-luoooddreeeiiiiii/ie revisiitineiiiiiiin dawe-inn-daa--aanneeeeeeeeeedeee www.deling- concern.
INTRODUCTION
Proprietary models have been proven to, perhaps inadvertently, learn from leaked benchmark data, raising questions about the reliability of closed-source models (Magar & Schwartz, 2022 Balloccu ct al. 2024). One particularly subtle form of contamination is data laundering. Distinguished from direct contamination, which define the scenario where model creates training-time shortcut by memorizing exact test samples present in its training corpus, data laundering is where test-set knowledge leaks student model during knowledge distillation via contaminated teacher, compromising evaluation integrity (Mansurov et al. 2025). While prior work highlighted this phenomenon, the prevalence, magnitude, and mechanisms of laundering remain largely unexplored. To assess the critical risk of this phenomenon, we ask: is data laundering a pervasive threat that undermines current benchmarking practices? We focus on classification tasks due two factors: first, it offers a relatively straightforward setup that is likely to provide reasonable approximation of the effects of laundering in broader scope of settings, from sequence generation to ranking; second, many modern applications of NI.P technology (from NER to WSD) are still explicitly framed as classification problems. Our extensive experiments on classification tasks suggest that data laundering is often much weaker than direct contamination and even mitigate some of its harmful effects. This work contributes a potential foundation for establishing safer empirical research environments.
First, we conduct a large-scale assessment across eight benchmarks to determine the prevalence and magnitude of data laundering. We find that significant laundering is a rare phenomenon: while it does occur, its effect model accuracy is substantially smaller than that of direct contamination, and in many cases the difference is not statistically significant. This initial finding suggests that knowledge distillation may indeed function effective decontamination method.

---

Published as a conference paper at ICLR 2026
Given that data laundering effects substantially weaker than those of direct contamination, investigate whether it is simply a watered- down form of direct contamination or a distinct phenomenon. Using sample-level analysis across the same benchmarks, to direct contamination predicts its sensitivity to laundering. Our analysis reveals only a weak correlation between the two; samples highly susceptible to direct contamination not necessarily the ones most affected by laundering. This suggests that laundering is indeed a distinct phenomenon.
examine whether a sample's 's sensitivity
Finally, we explore the conditions under which data laundering emerges. Controlled experiments show that systematically widening the train-test distributional gap increases the effects of laundering, suggesting causal connection. Consistent with these experiments, benchmarks with naturally larger trainttest gaps tend to exhibit stronger laundering effects, although the magnitude and statistical significance of the effect vary across datasets. Our findings indicate that both the characteristies of the benchmark and the degree of train-test shift influence the extent of data laundering, highlighting that its occurrence depends on benchmark -specffi factors such as dataset domain and the degree of train-test distributional shift, rather than being universal consequence of knowledge distillation.
In summary, this paper systematically disentangles the role data laundering in model evaluation. We argue that while the concern is valid, its practical impact is often minimal. Our findings indicate that laundering is generally limited in scope, substantially smaller than direct contamination, and tied to particular conditions such as train-test distributional shifts. By contextualizing these risks and identifying the conditions under which they arise, we provide a pathway for more responsible and reliable model evaluation including principled use of KD in the era of ubiquitous large models.
2 RELATED WORK
Data contamination in evaluation. A growing number of work has shown that benchmark integrity can be compromised when test material leaks into pretraining or fine-tuning corpora, artificially inflating scores without corresponding generalization. Early red flags already appeared with large web-scale LMs such as GPT-3 (Brown et al. 2020) and in corpora audits like C4 (Dodge et al. [021). while Magar& Schwartz (2022) provided a controlled, task-level analysis linking memorization to performance inflation. Subsequent studies proposed black-box and white-box detectors quided-instruucion "time travel" tests (Golchin & Surdeanu 2024) and distributional peakedness and documented practical challenges for closed models (OpenAl (003).
checks (Dong et al. Broader surveys and empirical audits emphasize that overlap can be subtle (paraphrases, partial spans, synthetic rephrasings) and uneven across benchmarks, motivating routine, benchmark-specifir contamination checks (Sainz et al. 2023) and calls for provenance transparency (e.g., to report train-test overlap: Zhang et al. (025). ). Recent work also targets modern LLM benchmarks directly, offering methods tailored to both open and proprietary models (Deng et al. 2024).
Knowledge distillation and data laundering. for compression and transfer (Hinton et al. 2015 Sanh et al. 2019). but also opens a distinct vector for leakage. Mansurov al. (2025) formally introduced data laundering showing that contaminated teacher (exposed to test data) can pass benchmark-specinf knowledge to a student trained only on clean data via KD, inflating evaluation without direct access the test set. However, their study had limitations: the experiments relied on bert-base-uncasse student trimmed down to just 2 layers, rather than using pretrained 2-layer model, making results difficult disentangle from near-random basclines. Additionally, the study did not compare laundering against direct contamination or systematically explore when it arises. As a result, the prevalence, magnitude, and mechanisms of laundering remain unclear, motivating our more systematic analyses. Complementary evidence from ranking distillation shows that even tiny teacher exposure (c.g., < 0.1% of training) can yield inflated student effectiveness, especially with pairwise/order-baseee jet al. 2024). Eecurity-orienter studies further show that KD can transmit non- bon-benign artifacts (e.g.. backdoor behaviors), particularly in data-free settings (Hong et al., (023). Together these results underscore procedural defenses such as transparent training histories and contamination-awe aware KD
Knowledge distillation (KD) is standard tool
objectives (Suresh Kalal
protocols (Zhang al. 2025)
2

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 EXPERIMENTAL SETUP AND DATA
Models and Distillation Process To isolate and measure data laundering, we use a controlled, two-stage process. Our setup involves fine-tuning eight models in total for each benchmark. First, we train teacher models using bert porrtbbaseunnasee (Devlin et al. 2019). A clean teacher (Telean) is fine-tur tie-ttned on the original training set, while a dirty teacher (Tdirty) is fine-tuned on training set contaminated with test data. Besides, to validate the generality our findings across different architectures, we also include modern large language models, namely Ll 2-1B (Grattafiori et al., 2024) and Qwen3-0. 0.. 6B (Yang et al. 2025). as teacher models. These models represent the current state-of-the-ar in lightweight LLMs.
Second, we use these teachers to distill knowledge into smaller student models (distilbert-bbs---nceedd (Sanh et al. (019) using soft-label distillation with forward KL divergence (Wang et al. 2024). Crucially, the distillation process for both student types (Selean distilled from Tetean and Sairty from Tairty)) is always performed using the original, clean training set. This ensures that any test-set knowledge is transferred exclusively via the teacher model, not through direct data exposure during the student's training.
To serve as a control group and contextualize the results, we also establish baseline models that share the same architecture as the students. A Clean Baseline (Belean) and Dirty Baseline (Bairty) are created by fine-tuning the student architecture directly on the clean and contaminated datasets, respectively.
Benchmarks and Contamination Protocol We selected eight public classification benchmarks, with diverse domains ranging from topic classification, sentiment analysis, and intent recognition to emotion detection, and NLI (details in Appendix B.2, Table 5p. This diversity ensures that our findings generalize across tasks with varying difficulty, number labels, and dataset sizes. For each benchmark, we create contaminated dataset train Bairty and Tairty with replacement strategy: we contaminate training data by injecting the full test set and removing an equal number of original training samples keep size constant, All experiments repeated with five different random seeds to ensure robustness.
3.2 EVALUATION METRICS
We employ set of metrics to quantify data leakage, distinguishing between those that operate on the entire benchmark and those that apply to individual samples. Let the test set be denoted by = {#1, #2, r. }. Metrics computed over capture the overall impact of data leakage on
model accuracy at the benchmark level, and sample-level metrics evaluate performance on individual examples Zt.
Sample-Level Leakage Scores To analyze leakage mechanisms at finer granularity, we measure how much each test sample 2; becomes easier or harder under different training conditions. The difficulty of sample x; under model M is defined
D(x M) = P( æ M),
that is, the probability of the model assigning the wrong label. Intuitively, higher D( M) means the model finds more difficult.
We then define two sample-level leakage effects:
Alauna (x4) D(r,, Suiny) D(ra, Sclean);
Acontam (Ix) (xx,, Bany) Dax,, Betean).
Here Atand captures how much student model changes when trained on dirty clean teachers, while Acontam captures how much a baseline model changes when directly trained on dirty vs. clean data. These sample-level scores provide natural way capture the effect of laundering and contamination on individual examples. In practice, we typically expect these values to be negative, since the presence of laundering contamination generally reduces the sample difficulty.
3

---

Published as a conference paper at ICLR 2026
Laundering vs. Contamination Correlation To assess whether data laundering and direct contam- ination are mechanistically related, we compute benchmark- correlation from the sample-level scores. For given test set C, first construct two vectors of leakage effects:
lannd (21), ..... laund (xn))
comam (21), (21),.... .... Acontam (2:n)
We then calculate the Pearson correlation coefficient
cov(I,
r(C)
(I)
σ 0e
This metric is advantageous as it is scale-invarians allowing us to compare the directional agreement of the two phenomena across all samples, irrespective of the absolute magnitude of their effects.
4
RESULTS AND ANALYSIS
4.1
KNOWLEDGE DISTILLATION AS AN EFFECTIVE DECONTAMINATION TECHNIQUE
200
Teacher
Tescher Dirty
Baseline
33
Beceline Birty
Student
Student Dirty
Tr
50
Zinewgrroup
Performance clean and contaminated benchmarks for Teacher, Baseline, and Student models. Bars show clean (lighter) and contaminated/dirty (darker) accuracies, with AAcess values annotated. The statistical significance of the differences are marked with "p<0.05, "pc0.005, *** p<000011.
Figure 1:
establish an upper bound on potential data leakage by comparing the aggregate impact contam- ination using benchmark-lewe metrics. Specifically, contrast the gains from direct contamination in clean and dirty baseline (B) and teacher (T) models with the gains from data laundering clean and dirty student (S) models. For model M and benchmark C, the performance gain due test-set leakage at training time defined as
AAcaM A.cc C) Acc AccCeeMnnn: C).
(2)
When M = B or M = T, Equation (2) measures the direct contamination effects, whereas for M = S it measures data laundering gains.
Figure presents the performance gains over the baselines for all models (the full results are reported in Appendix C T8)le ()). Comparing the gains of baseline and student models allows quantify the relative impact of data laundering on observed performance improvements, and contrasting the direct contamination gains of baseline and teacher models highlights the impact of model capacity
The results reveal clear and consistent pattern: directly training on cest-set-contaminated dataset results in substantial and highly significant performance gains across all eight benchmarks for all Teacher and Baseline models. Performance gains for the Baseline models range from 4.89% on emotion 25.66% on tweet_sentiment confirming the well-dooummnted effects of direct data contamination and providing crucial reference point for evaluating the impact of knowledge distillation.
An exception SNLI, where the performance gap between clean student and the clean baseline unusually large. This result may be explained by two factors: (i) all models were trained for only epochs without early stopping to ensure fair comparison with the teacher and baseline, which may have left the student undertrained when relying solely teacher signals; and (ii) SNLT I is inherently more challenging than typical classification tasks, making it harder for the student to achieve strong performance under distillation.
4

---

Published as a conference paper at ICLR 2026
By contrast, the gains observed due to data laundering (AAces) are noticeably smaller, highlighting the mediating effect of knowledge distillation. For example, on tweet sentiments the Baseline's 25.66% gain is reduced 3.25% after distillation through contaminated Teacher. Similarly, 20newsgroups, an 11.91% % direct gain shrinks to just 1.42%. This trend is consistent across all benchmarks: distillation acts strong bottleneck, significantly mitigating the performance inflation caused by direct contamination. Overall, these results suggest that knowledge distillation, rather than solely propagating leakage, may act as effective decontamination mechanism, substantially mitigating the performance inflation caused by direct contamination.
The gains from data laundering are clearly more modest than those from direct contamination, with significant increases on datasets like agnews (0.65%) and sreet_sentimens (3. (3.25%). However, for three benchmarks (banking77, rotten tomatoes, and SNLI) the difference performance not statistically significant. Furthermore, to assess the practical significance of these gains, we also computed bootstrap confidence intervals (see Appendix B.4) which provided further confirmation of the observed effects.
This detailed observation allows us to refine our perspective. While the concern about data laundering is not unfounded, as the phenomenon does occur, its practical impact is the exception rather than the rule. a rare and mild effect, with a smaller magnitude and limited influence on overall model evaluation.
Robustness across Distillation Strategies To examine the robustness of our findings, we further extended our analysis across diverse distillation objectives, including Soft Forward, Soft Reverse, and Hard-label distillation under both pure and mixed loss formulations. We observe modest variations in vulnerability: mixed objectives and soft labels can slightly increase the impact of data laundering. However, the overall pattern remains consistent: knowledge distillation acts as an effective decontamination filter in classification tasks. Across all investigated strategies, statistically significant data laundering remains an exceptional and -w-aaniitude phenomenon compared to direct contamination. We provide detailed breakdown of these results and the associated trade-offs in Appendix C.2
Generality across Modern Architectures We further validated our findings using Llama-3.2-11 BB and Qwen3-0.6B as teachers, selecting four representative tasks from our benchmark suite: AGNews, Emotion, Rotten Tomatoes, and Tweet Sentiment. In these experiments, the student model remains DistilBERT trained with Soft Forward distillation (a = 1, T = 2.0), ensuring that any differences reflect the teacher's 's influence rather than changes to the student architecture. Detailed experimental setups and the full set results can be found Appendix C.3 (Table 12).
Figure@present presents the performance of the student under Llama and Qwen teachers, showing both the clean and contaminated teacher models, along with the corresponding student outcomes. While the I.LM teachers achieve near-perfect accuracy on some contaminated benchmarks (e.g., Tweet Sentiment and Rotten Tomatoes), student gains from direct contamination remain minimal, consistent with the observations from BERT-based teachers. For instance, while the Llama-3.2 222 teacher gains 25. 55.80% on Tweet Sentiment, the student gain is limited to 1.91%, underscoring the informational bottleneck imposed by distillation.
These results highlight that switching to a larger. more capable teacher has minimal effect on student contamination, reinforeing our conclusion that KD effectively limits the propagation of memorized data. However, we emphasize that these preliminary experiments are to be interpreted with caution. likely that the test sets overlap with the I.LMs' pretraining data, which inflates observed gains and undermines our attempts to keep our experiments under controlled scenario.
Nevertheless, as shown in Table the choice of teacher produces relatively small differences in student models under our "clean student" setup. Even if we treat Llama or Qwen as effectively dirty teachers (given their likely exposure to our benchmarks), comparing them to our guaranteed clean teacher constitutes clean-vs-dirty scenario. Under this interpretation, the lack of substantial gains in students distilled from LLM teachers is consistent with our earlier conclusions. Data laundering has limited impact and KD can serve buffer, even in the presence of strong teachers.
5

---

Published as a conference paper at ICLR 2026
LLAMA
QWEN
Tcadee Clzan
Teacher Dity
120
Daseinn Clean
Raysline Dirty
Studert Clann
Dity
Neforunna
20-
75
65
Aguive
Emokiin
Tomatoes
Seationnt
Appess
Boaanee
Sectioann
Figure 2: Performance on clean and contaminated benchmarks for LLM Teachers, Baseline, and Student models. The teachers used are Lama-3.2-1B IB and Qweb3-0.6B. Bars show clean (lighter) and contaminated/dirfy (darker) accuracies, with AACCM values annotated. The statistical significance of the differences are marked with pc0.05, Pcc0.005, p<0.001.
Teacher Model
agnews
emotion
roffen tomatoes
[weet sentiment
BERT-base-mnessd LLAMA3.2-1B
93.10
83.77
66.99
92.81 (-0.29)
84.15 (+0.38)
(+0.14)
QWEN3-0.6B
92.82 (-0.28)
84,22 (40.45)
(+0.1
Table 1: Performance comparison of the clean students (Ssaft. when trained with different teachers. The numbers in parentheses indicate the gain (or loss) relative to the BERT student.
4.2 LAUNDERING AND CONTAMINATION: A TALE OF TWO MECHANISMS
Having established that data laundering is a rare and much weaker effect than direct contamination across diverse architectures and objectives. we now turn to dissecting the mechanism itself. To conduct a granular, sample-level analysis in strictly controlled environment, we return to our primary experimental configuration: a BERT-based teacher with Soft Forward distillation. Under this canonical setup, a crucial question arises regarding the nature of the phenomenon: is laundering merely a weaker, scaled-down version of direct contamination, or is it a distinct phenomenon with its own underlying mechanism? If it were simply "contamination-lite we would expect the samples most affected by both phenomena to be highly correlated.
20newsgroups
agnews
banking77
emotioo
sentiment 0.31
r(C)
0.30003)**
0.32(02)
0.13(08)
0,26(12) 0.30002) 0.12(06)
-0.03(17)**
Table 2:
Benchmark correlations hetween laundering and contamination effects. Shows baseline accuracy (Beean) and Pearson correlations of sample-level effects. Statistical significance was assessed using hootstrapping- based tests, with detailed procedures provided Appendix B.4
In Table 2 we show the Pearson correlation between the sample-level laundering effect and contam- ination effect scores. We find that on the benchmarks most susceptible laundering, agnews and tweet_sentiment the correlations are as small as 0.32 and 0.31, respectively. While statistically significant, these values are far below the commonly accepted threshold of 0.7 for strong relation- ship (see e.g., Rickert et al.] 2023 Kjell al. 2022), indicating only weak linear association. The connection weaker still on other benchmarks, and even becomes slightly negative for snli.
This weak correlation suggests that the two phenomena impact samples differently. A sample that is highly vulnerable to being "memorized" through direct training is not necessarily the same sample whose knowledge is indirectly transferred through teacher model. This observation motivates deeper exploration into the nature of these mechanisms. To explore this further, we conduct a granular, sentiment benchmark, where the laundering effect most
sample-level analysis the pronounced. F visualizes the laundering and contamination effects for each test sample, sorted by their difficulty as perceived by both the clean baseline and clean student models.

---

Published as a conference paper at ICLR 2026
Sumpic meaiee onion
prfccry
(a) Samples sorted by clean baseline difficulty
(b) Samples sorted by clean student difficulty
benchmark, with samples sorted by difficulty. The contamination effect (orange) shows a strong, monotonic downward trend as samples become harder. In contrast, a laundering effect (blue) is more dispersed, has a much weaker trend, and is less correlated with initial sample difficulty.
Figure 3: Laundering and contamination effects on the tweet_sentinee
By sorting samples by the difficulty perceived by the clean model, we visualize that some samples are more inherently prone to contamination than others: those that the clean model finds more difficult have a larger potential for improvement when seen during training. As result, the contamination effect (in orange) exhibits a strong and relatively monotonic downward trend-a more negative Acontam on harder samples. This an expected signature of direct test-set exposure. In stark contrast, the data laundering effect (in blue) does not share this strong monotonic relationship with sample difficulty. overall magnitude is smaller, its trend is weaker, and exhibits significant volatility, with benefits appearing for both difficult and some easy samples.
Together, the weak sample-leve correlations and non monotonoe laundering trends show laundering not "scaled-down contamination" but distinct mechanism. Instead, it is an independent mechanism triggered by different conditions. This discovery leads to the conclusion that data laundering, when occurs, is more elusive mechanism than direct contamination and likely possesses its own unique set of enabling conditions.
5 DISCUSSION: THE ROLE OF DISTRIBUTIONAL GAPS
Our analysis in the previous section established that data laundering is a rare, mild, and mechanically distinct phenomenon from direct contamination. This prompts the next logical question: what specific characteristics of a benchmark make it more susceptible to this elusive effect? Our results revealed the impact of data laundering to be highly benchmark-speciffr with datasets such as se_senntieen timent exhibiting comparatively more significant effects. This naturally leads us to hypothesize that some intrinsic property of these datasets may be at play.
Jaccard
TF-II
Avg Lim Sim
Average Max Semantic Sim
Average Pallemm Conformity
Dataset
Macro
Micro
Macro
Micro
Micro
Macro
agnows
0.1686
0.1463
0.9881
0.9661
0.9984
0.9965
0.6682
0.6554
0.4942
0.3468
tweet, sentiment
0.0887
0.0733
0.6310
0.6371
0.7115
0.7681
0.5102
0.4913
0.4437
0.4523
20newsgrops
0,1787
0.1513
0.9848
0.8559
0.9921
0.9599
0.5985
0.5709
0.4871
0.5431
emotion
0.0941
0.0666
0.9820
0.072
0,9984
0.9877
0.6400
0.5954
0.5504
0.5622
imdb
0,2218
0.2116
0.9978
0.9961
0.9994
0.9990
0.6919
0.6837
0.6069
0.6707
rotten, tomatoes
0.0702
0.0606
0.9198
0.8609
0.9987
0.9975
0.6331
0.6229
0.5920
0.5958
snli
0.9906
0.9877
0.9979
0.9975
0.7011
bunking77
0,2874
0.2217
0.9810
0.9103
0.9963
0.9887
0.8985
0.8940
0.7520
0.8627
Table 3: Similarity metrics between the test set and the training subsets. "Micro" refers to the global similarity, calculated across all samples without referencing labels. "Macro" refers to the unweighted average of similarities computed per-label basis. Benchmarks vulnerable data laundering, such as tweet_sent irreet_sentmmet, happen to appear lower similarity scores.
To investigate this, we first characterize the intrinsic relationship between the training and test sets in model-agnostic way. We compute suite of similarity metrics, all normalized to the range [o, 1], to help identify inherent data properties that might make a benchmark susceptible to leakage. The metrics used Jaccard Similarity, TF-IDF Cosine Similarity, Average Embedding Similarity,
7

---

Published as a conference paper at ICLR 2026
Average Max Semantic Similarity, and Average Pattern Conformity. The detailed mathematical formulations for these metrics are provided in Appendix B.5
An analysis of these metrics across the benchmarks (detailed in Table revealed notable pattern. The benchmarks most affected by laundering (c.g., tweet_sentiment consistently exhibit lower similarity scores across several metries. This indicates larger distributional gap between their training and testing sets. This observation provides us with a plausible hypothesis: data laundering is more likely occur, and its effects are more pronounced, when there a significant distributional distance between a benchmark's 's training and test sets.
The intuition behind this is that when teacher model is contaminated with test data that deviates from the training data's 's dominant semantic patterns, it is exposed to alternative, test-specific regu- larities. These regularities different from those emphasized in the training distribution can then be systematically learned by the teacher. Crucially, such patterns may be particularly advantageous for the test set, and can subsequently be passed on to the student during distillation. Based on this hypothesis, we designed series of controlled experiments to systematically validate this relationship.
5.1 EXPERIMENTAL DESIGN
To test our hypothesis rigorously, we designed a controlled experimental setup to systematically vary the train-test distribution gap. This section outlines the creation of our stratified datasets and the experimental protocol.
Creating Controlled Distributional Gaps The core of our methodology involves partitioning the training data of the emotion and rotten_tomatoes benchmarks into distinct subsets. This is achieved through stratified splitting process: for each class, we first identify its test-set centroid, and then partition the training samples belonging to that class into five equal-sized quintiles based on their semantic similarity to this centroid. These quintiles are then aggregated across all classes to yield five global training sets, denoted Levels 1 through 5. Level 1 contains samples most similar to the test set (smallest gap), while Level 5 contains those least similar (largest gap). This method successfully creates the intended gradient of distributional gaps. The effectiveness of this partitioning is validated by systematic decrease in cosine similarity from Level to 5, as well as a consistent monotonic decrease across five other similarity metrics (see Appendix B.6 for detailed visualizations).
Training and Contamination Protocol For each of the five data levels, we conduct a consistent training and distillation protocol. We train a clean teacher (Telemn) on the original training quintile and a dirty teacher (Tany) on its contaminated counterpart. We use an add mode (as opposed to replace mode) for contamination in this setup: because each training level is significantly smaller than the original dataset, this approach ensures the model has sufficient data for robust training and mitigates the impact of the reduced training set size. Knowledge from both teachers is then distilled into respective student models, Selean and Sairty- The distillation process itself always uses the clean training data of that level, isolating the transferred knowledge as the primary variable.
5.2 RESULTS: LAUNDERING EFFECT INTENSIFIES WITH WIDER DISTRIBUTIONAL GAPS
With the controlled gaps established and verified, we now turn to the central result of our experiment. As shown Figure our findings demonstrate clear relationship between the train-test distributional gap and the manifestation of data laundering. Specifically, we observe that as the distributional gap widens, the data laundering effect becomes statistically more significant and pronounced, and in some cases, increases in magnitude.
Direct Contamination Effect Remains Stable First, we observe that for both the emotion and rotten_tomatoes datasets, significant direct contamination effect is present from Level 1 to 5, evidenced by the performance differences between the dirty and clean teachers, as well as between the dirty and clean baselines. However, this performance gap does not show significant fluctuation or a clear trend as the distributional gap widens. This can be observed visually in Figure 4 where the gaps between clean and dirty models (for both teachers and baselines) remain largely stable across the
8

---

Published as a conference paper at ICLR 2026
0,030:
3,0071
1.0001
0.078:
01443
"p
5.0311
1.60
0.002:
0002022
0.000
5.0311
10601
0220
010301
Sued
Se602
Send
(a) emotion accuracies
(h) emotion significance heatmap
button Gup
0.000:
1,030
1.0033
0.3437
00215
0.834
1,0000
2 0000
0.0282
60002
P
Seud
Seed2
Send Random Seud
Sand
Seed
(c) wwwtro_nttaatetteeeee tomatoes accuracies
(d) Tratten_ttmatooe significance heatmap
Figure 4: Data laundering effects across controlled distributional gaps for emotion and rotten_ tomatoes. Left panels: Model accuracy comparison, where the performance gain for the student model (red bars, AAccs) represents the laundering effect. Right panels: Heatmaps of p-values testing the significance of the laundering gain over 5 random seeds. The increasing dark- ness from Level 1 indicates that the statistical significance of the laundering effect becomes more pronounced as the train-test distributional gap widens, even if the absolute magnitude of the gain varies. Statistical significance (xp < 0.05, * < 0.01, *p < 0.001) was assessed using mootstrapping-basee tests.
levels. Our experiment thus indicates that, under this setup, the magnitude of the direct contamination effect not significantly correlated with the distributional gap between the training and test sets.
Analysis of the emotion Benchmark: Increased Statistical Certainty We first examine the emotion dataset. As seen in the bar charts of Figure 4 a). the absolute performance gap between clean and dirty students (AAces) remains relatively small across all levels. However, the significance heatmap in Figure bb) reveals crucial trend: the p-values systematically decrease as we move from Level 1 (smallest gap) to Level 5 (largest gap). This indicates that while the magnitude of the leakage may not drastically increase, the statistical certainty of its existence strengthens. other words, large distributional gaps transform laundering from rare and inconsistent occurrence into consistent, statistically significant phenomenon, even if the resulting accuracy gain is modest.
Analysis of the totetteesmttoss Benchmark: Growth in Both Significance and Magnitude The rotten_tomatoes tomatoes benchmark exhibits an even more pronounced pattern. Consistent with the trend seen in emotion, the statistical significance of the laundering effect increases as the distributional gap widens (Figure 4 d)), with p-values consistently dropping to highly significant levels (p < 0.01 or < 0.001) at larger gaps. Moreover, unlike emotion, this benchmark also shows visible widening of the performance delta between clean and dirty students the level increases (t))r This suggests that for certain datasets, larger distributional gap not only increases the likelihood of laundering occurring can also increase impact on model performance.
Summary and Implications Our results identify the train-test distributional gap as a key factor driving data laundering, increasing its statistical consistency even where absolute magnitude varies.

---

Published as a conference paper at ICLR 2026
In contrast, direct contamination shows no such correlation. This finding reveals trade-off in benchmark design: the distributional distance required to evaluate generalization may inadvertently increase laundering risks. Consequently, argue that the rigid single-test-set paradigm is insufficient. Instead, employing multiple test sets at varying distributional distances offers a more robust approach, simultaneously assessing generalization power and resilience to data laundering.
Practical Guidelines for Benchmark Release We propose a rigorous two-step curation protocol based on these findings: (1) Diagnostic Gap Measurement: Prior to release, curators should quantity train-test shifts using the lexical and semantic metrics detailed in Appendix B.5 Consistently low
similarity scores serve as a diagnostic flag for high laundering susceptibility. (2) Distance- Aware
Evaluation: If a large gap is detected, we recommend releasing multiple test splits positioned at varying distributional distances. This enables a systematic audit of knowledge distillation systems to distinguish between true robust generalization and artifacts of data laundering driven by specific distributional shifts.
CONCLUSION
This paper set out to investigate the severity of data laundering and its implications for the integrity of classification evaluation. Our comprehensive investigation across eight benchmarks offers reassuring, if nuanced, conclusion: concerns of data laundering pervasive threat appear largely overstated. Instead, we find that knowledge distillation appears be a promising decontamination technique, dramatically attenuating the performance inflation caused by direct test-set exposure.
While distillation acts as a strong buffer, not perfect one. We confirm that residual leakage can occur, but these instances of significant laundering are the exception, not the rule. Crucially, our analysis reveals that this leakage is not simply a diluted form of direct contamination but mechanically distinct phenomenon. We identified the train-test distributional gap as key driver, a hypothesis confirmed through controlled experiments where systematically widening this gap induced significant laundering effect.
In short, knowledge distillation, far from being a liability, is a robust defense against test-data leakage in classification settings. The rare, benchmark-sye -speciffc instances of laundering not indictment of the method, but predictable consequence of large distributional shifts between training and test data-a factor that benchmark designers may need to consider. Our findings thus help clarify the risks associated with distillation and provide path toward more reliable and responsible model evaluation.
7 LIMITATIONS AND FUTURE WORK
Limitations Our work is primarily constrained by its focus on BERT-family encoder-only classifi- cation models, lcaving the effects on larger-scale models and different student architectures, such decoder-only LMs, unexplored. Besides, the analysis laundering in generative contexts, where distillation and contamination evaluation strategy can be more diverse and complex, even in scenarios where teachers generate synthetic data (data-free KD), remains an open question. Finally, our experiments used only English datasets, so the findings may not generalize multilingual or domain- speciffc scenarios.
confined classification tasks; data
Future Work These limitations suggest new research avenues. Future work should extend our work to broader architectures, training regimes and tasks. key step require extensive efforts will be the development of appropriate metrics for generative models: in the present state, our study does not addres text-generation systems, which an import point to cover given the the current research landscape. It is also crucial to examine data laundering on multilingual and domain-specific benchmarks to test the generality of our findings and to further probe the mechanistic distinctions between laundering and direct contamination. While our current full test-set replacement constitutes a controlled worse-case scenario, studying contamination under training regimens that resemble how leakage occurs in the wild (typically, partial, noisy, or indirect) remains an important direction to study. Besides, a particularly promising direction involves leveraging KD for decontamination use, and even act diagnostic tool to infer the contamination level of teacher models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This project is funded by the AI-DOC program hosted by Finnish Center of Artificial Intelligence (decision number VN/3137/202--KKM project, co-funded by the Digital Europe Programme under GA no. 101195233. The authors wish to acknowledge CSC IT Center for Science, Finland, and LUMI supercomputers, owned by the EuroHPC Joint Undertaking, for providing computational resources.
This research is also supported by the OpenEuroLLM
REFERENCES
lbtihel Amara, Nazanin Sepahvand, Brett H. Meyer, Warren Gross, and James J. Clark. Bd-kd: Balancing the divergences for online knowledge distillation, 2024. URL https:::s///axxi..com org/ abs/2212. 12965
Simone Balloecu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source L.MMs. In Yvette Graham and Matthew Purver (eds.). Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 67-93, St. Julian's Malta, March 2024. Association for Computational Linguistics. doi: 10. 112.18663/11/2222.ccccooor
URL
https://aclanthooooyy.rr///// eacl--lng. 5/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Chunyuan Deng, Yilun Zhao, Xiangru Tang. Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Pupers), pp. 8706-87 19, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18653/81122224..... www.:l2oooaaal-oooo......o 482. URL pattp:://aclantholo.s. org. 2024
naacl -lng. 482/
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4188 Minneapolis, Minnesota. June 2019, Association for Computational URL /RLlnttt////aalanttoooo wwwpanttwwooooooooooooa/e www.ooggg//oN. 1423/
Linguisties. doi: 10. 11.18655/11/N1--1222
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Mar- garet Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study the colossal clean crawled corpus. In Proceedings of EMNLP, Punta Cana, Dominican Republic, 2021. Associa- tion for Computational Linguistics. URL htttps://aclanthhhe:
org/ 220g//211...m emnlp-main camp-mmai.... 98/
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. Findings of ACL. pp. 12039 12050, 2024. URL https://aclaathhoogy.orrrg 2024. findings-aeee acl. 716
pdf
Shahriar Golchin and Mihai Surdeanu. Time travel in LLMss: Tracing data contamination in large language models. In Proceedings of ICLR, 2024. URL http:/ttpps//geeerrrvevv///////r
2Rwg6c3tvr
Aaron Grattafiori et al. The llama 3 herd of models, 2024. URL URU.ttpp://arxvv.or/////// 21783
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: /50.vv 100..0533332311 2015.
11

---

Published as a conference paper at ICLR 2026
Junyuan Hong, Yi Zeng, Shuyang Yu, Lingjuan Lyu, Ruoxi Jia, and Jiayu Zhou. Revisiting data-free knowledge distillation with poisoned teachers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 13199-13211 PMLR, 23 Jul 2023. URL htt.hhtts//prrcceennggst mlr press/ v262/hong23c.h httm html
Oscar N. E. Kjell, Sverker Sikström, Katarina Kjell, and H. Andrew Schwartz. Natural language ana- lyzed with ai-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy. Scientific Reports, 12(1), March 2022. ISSN 2045-2322, doi: 10.1038/s41 -98--0220775200 URL http: //dx. doi org/orgg/// 1038/ -415944022-99909
Inbal Magar and Roy Schwartz. Data contamination: From memorization exploitation. In Proceed- ings ACL (Short). pp. 157-165, Dublin, Ireland, 2022. Association for Computational Linguisiics doi:
18/
Jonibek Mansurov, Akhmed Sakip, and Alham Fikri Aji. Data laundering: Artificially boosting benchmark results through knowledge distillation. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8332-8345, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979 3-89976-221-0 doi: 10.18653/v 1/201/2222222 aww.:/2255.----oo--441.. 407. URL /RRL/tttss///cllattooooyo.oo/ hww.2/222..1111..4444111
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.087774 2023. URL https://arxiv org/abs/2303.0m
Carolin A. Rickert, Manuel Henkel, and Oliver Lieleg. An efficencyyddrieen, correlation- feature climination strategy for small datasets. APL Machine Learning, 1(1):016105, 02 2023. ISSN 2770-9019. doi: N0.1063/5.011 182077
1063/5. 001.011110 0118207
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need measure LLM data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776-10787, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18.138/3/v 122553//1/222.... findings-emnlp.725 URL https::/aclannholoogy.. wwwptthhoooggy..o//222222 11122.ffinding findings-emnle 722/
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100 2019.
Vishakha Suresh Kalal, Andrew Parry, and Sean MacAvancy. Training on the test model: Con- tamination in ranking distillation. In arXiv preprint arXiv:2411.02284 2024. URL https
//arxiv.org/abs// http:/abbb22111.222222
Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence-based knowledge distillation for unlearning personal information in large language models, 2024. URL
https://arxie wttp://w.wr..b////a/aooooo 240e 01983
Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang. Zhe Zhao, and Ngai Wong. Rethinking Kullback-Leibler divergence in knowledge distillation for large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 5737-5755 Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://acclnthont wrw.oology.oogg/g/eooooooo .rg. 2025. wwwwnooonnnn..ee main. 383/
An Yang et al. Qwen3 technical report, 2025. WRRLhttps:: http:///aawww...//////////5//// 09388
Andy K Zhang, Kevin Klyman, Yifan Mai, Yoav Levine, Yian Zhang, Rishi Bommasani, and Percy Liang. Position: Language model developers should report train-test overlap, 2025. URL http:////peeeeeeeeeeeee/errrr/rr
12

---

Published as a conference paper at ICLR 2026
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference Computer Vision (ICCV), pp. 19-27, 2015. doi: 10.1 10911C
A STATEMENTS
A.1 ETHICS STATEMENT
The datasets used in this work do not involve any sensitive or personally identifiable information, nor do they raise copyright concerns. Based on the experiments we conducted, we did find evidence of systematic bias against different genders, languages, regions. To the best of our knowledge, the experiments reported this paper don't raise ethical concerns.
A.2 REPRODUCIBILITY STATEMENT
We committed ensuring the reproducibility of results. All code, data, and preprocessing scripts will be released publicly after the anonymous review period. Detailed hyperparameters, hardware specifications, and other experimental settings are documented in Appendix Together, these materials should allow independent researchers to fully reproduce our results.
A.3 LLM USAGE DISCLOSURE
Large language models (LLMs) were used in a limited capacity during this work. First, Al tools were employed for minor English polishing, such as improving grammar and selecting more accurate word usage to convey ideas precisely. Second, Al tools were occasionally used to assist in debugging code during experiments. All scientific ideas, experimental designs, and contributions remain the authors' own.
B EXPERIMENTAL AND IMPLEMENTATION DETAILS
This section provides supplementary details on our experimental setup, models, datasets, and imple- mentation.
B.1 MODEL SETUP
Our experimental setup involves comprehensive suite of models, detailed in Table 4 A key principle of our design is the direct comparability between baseline and student models: they share the same architecture, training hyperparameters, and number of training samples. The only distinction lies in their supervisory signal- baselines learn from ground-trutt labels, while students learn from teacher's outputs.
To fully explore the characteristics of each distillation method, we deviate from the standard paradigm. Specifically, we investigate setups both with and without the cross-entropy loss term on ground-truth labels for student training. This is controlled by weighting coefficient, a. When a = 1. students learn exclusively from the teacher's supervisory signal. This ensures that any observed performance gain in dirty student is attributable purely to the laundered knowledge from the teacher. When 0.5, the student learns from balanced mix of the teacher's signal and the ground-truth labels.
For the sake of brevity, we primarily present the results from the Soft Forward distillation with = the main body of the paper. However, our experiments comprehensively cover all three core distillation methods: (i) soft-label distillation with forward KL divergence (SoftFwd), (ii) soft-label distillation with reverse KL divergence (SoftRev) (Amara et al.) 2024), which uses an alternative divergence measure, and (iii) hard-label distillation (Hard) (Hinton al. 2015). Each distillation method is evaluated in settings both with and without the Ground Truth Cross- Pross-Entropy loss. Detailed results across all these configurations be found in appendix
13

---

Published as a conference paper at ICLR 2026
Description
Training Data
Supervisory Signal
Baron
Clean Baseline
Clean
Ground-tre trath labels
0
Bairts
Dirty Baseline
Contaminated
Ground-trati labels
0
Shoud
Student from Tites (pure)
Clean
Hard labels from Takon
Stndent from Varny (pure)
Clean
labels from Tarty
Goa (o
ghard
Tires (mrixed)
Hard label from Zasan
Cesta ofs.)) Com(k.o(s..))
drrig
Student from Tarny (mised)
Clean
GT Hard labels from Tacsy
0.5 Costa.r((.)) 0.5 Cost8.ota.))
0.5
Student from Takres (pare)
Clean
Soft lahels from Jalenn
dart
Tatry (pure)
Tarty
KL(a(a//e) [[aa/r))
Student from Titess (mixed)
Clean
GT Soft lahels from Tates
Coz(g.o((s)) 0.5 KL(a(a/)) (o(s//))
Studene from V'drrp (mixed)
Clean
GT Soft lahels from Vorio
0.3 (esfa.n(sa]) 0.5 (L(n()/))) [o(s//)) 0.5
Student from Tares (pure)
Clean
Soft labels from Talen
KL(a(s/T)) [| o(a//r)
Studen: from Tairy
Clean
Saft labels from Tasty
KL(o(s./T) o(au/)))
Sa
Student from Titees (mixol))
Clean
GT Sonl labells from Titess
0.3 Costs.n(ss))
(r) [[
Student from Titrip (mixed)
Clean
GT Soft labels from Tamig
(csla.o(s.) 0.5 (L(a(s//)) [[((a/H))
Table Overview of the models in our experimental setup. Telean and Tairty denote the clean and dirty teachers. 2m, are the logits from the main model, student, and teacher, respectively. o the softmax function, y is the ground-truth label, Dr temperature. Hard distillation uses Cross-Entrope (CE) loss. Soft distillation uses KL Divergence; forward KL is "mean-secke seeainse while reverse KL is "modc-secking" (Wu et al.) 2025). The a parameter controls the weight of the ground-truth loss term; indicates pure distillation, while a = 0.5 indicates a mixed objective.
teacher's hard prediction, and 7 the
B.2 DATASETS
We eight public classification benchmarks, detailed in Table 5 We adopt BERT-base-uncased for teachers and DistiBEERT-bass-uucaaedd for students/baselines.
Benchmark
Task Type
Classes
Original Train Size
Original Test Size
Train Subset Ratio
Effective Train/ Test//Tsstte Ratio
Dnewsgroops
Topic Classification
agnews
Topic Classification
120,000
7,600
0.1
1.58
77
Insent Classification
77
10,003
3,080
1.0
3.25
emotion
Emotion Classification
16,000
2,000
1.0
$.00
indl
Analysis
25,000
rotten fomatoess
Sentiment Analysis
1,066
shii
Natural Languge Inference
$50.152
9,824
D.1
5.60
tweett sentiment
Sentiment Analysis
45,615
12.284
0.5
1.86
Table Details of the benchmark datasets used in our experiments. The Train Subset Ratio adjusts the training set size control the relative influence of training versus injected test data.
Benchmark
Classes
Original Train Size
Size per Stratified Quintile Original Test Size
Effective Train/ Test Ratio
emotion
6
16,000
3,200
2,000
1.60
rotten_ tomatoes
8,530
1,706
Table 6: Dataset details for the controlled distribution gap experiments. The training set for each benchmark was partitioned into five stratified quintiles based on semantic similarity to the test set. The table shows the resulting size of each quintile and the corresponding effective train/test ratio.
B.3 HYPERPARAMETERS AND COMPUTATIONAL RESOURCES
The same set of hyperparameters was used for training all baseline, teacher, and student models to ensure fair and controlled comparison. We trained all models for fixed number of epochs and did not use a development set for early stopping. The specific hyperparameters are detailed in Table 7
We utilized NVIDIA A100 GPUs for the standard BERT-based experiments in Section 4 The LLM distillation experiments, as well as the controlled distribution gap studies Section were conducted on AMD MI250X GPUs.
14

---

Published as a conference paper at ICLR 2026
Hy pepepaarreeeer
Value
Leuriing
Batch Size
32
Training Epuchs
Distillation Temperature
2.0 (for soft distillation only)
Max Soquence Length
(512 for IMDB & 20newsgroups)
Random Seeds
1.42. 86, 358. 1024
Table 7: Hyperparameters for all model training.
B.4 SIGNIFICANCE TESTING DETAILS
To ensure the reliability of our findings, we employed bootstrapping based statistical tests. The detailed procedures for assessing the significance accuracy gains and correlation coefficients are outlined below.
Accuracy Gains (Clean vs. Dirty) To determine if the accuracy of a "dirty" model was significantly higher than its "clean" counterpart, we used one-sided paired bootstrap test. For each of the five random seeds, we first drew 10,000 bootstrap samples (with replacement) from the test set predictions of the clean and dirty models. Then, for each bootstrap sample, we calculated the difference in accuracy between the dirty and clean models. Finally, the p-value was estimated as the proportion of bootstrap samples where the clean model's accuracy was greater than or equal to the dirty model's accuracy. To maintain conservative assessment, we report the maximum p-value observed across the five random seeds for each benchmark comparison. In addition to significance testing, we quantified the uncertainty of the performance gains. We computed 95% confidence intervals (CI) for the accuracy difference using the bootstrap percentile method. Specifically, the lower and upper bounds of the CI correspond to the 2.5th and 97.5th percentiles, respectively, of the distribution of accuracy differences obtained from the 10,000 bootstrap samples. our results, we report the average 95% CI across the five random seeds to represent the stability of the observed gains.
Correlation Coefficients To confirm the stability and significance of the Pearson correlation coefficient between the laundering effect and the contamination effect, we used two-sided bootstrap test. For each random seed, we performed 10,000 bootstrap resamples of the test set samples. For each resample, we re-calculated the Pearson correlation. The p-value was then derived from the distribution of these bootstrapped correlation coefficients to test the null hypothesis that the true correlation is zero. We report the maximum p-value across the five seeds.
B.5 MODEL-AGNOSTE DATA CHARACTERISTICS
To characterize the relationship between the training and test sets in a model-agnostic way, compute a suite of similarity metrics. These metrics, all normalized to the range jo, 1]. help identify inherent data properties that might make benchmark more susceptible to leakage. Let Cuain and Creat denote the training and test corpora, respectively. For some metrics, the similarity a direct comparison between corpora, while for others, it is aggregation of sample-level calculations.
Jaccard Similarity: Measures lexical overlap based the set of unique n-grams (Ng) present in each corpus. This is a direct corpus-level comparison.
NN (Curan)) n. N (Cee))
Jaccard Cres) Ng (Curain) U (Crsa)
TF-IDF Cosine Similarity: Measures similarity by comparing the centroids of the corpora in the TF-IDF vector space. First, each corpus C is represented by its mean TF-IDF vector, (C), which is an aggregation of individual sample vectors.
(C) = IC|
(ana(())
#EC
The final similarity is the cosine distance between these two mean vectors.
Sim upr (Criii Crea) cos_sim Vinar(Curin)) Frar (Crest)
15

---

Published as a conference paper at ICLR 2026
Average Embedding Similarity: Similar to TF-IDF, this metric computes the cosine similarity between the mean Sentence- Meanncee-RTTTee embedding vectors of the corpora. The mean embedding vector for corpus, E(C), is derived by averaging the embeddings of all its samples.
aC)
emb(z)
|C|
The similarity is then calculated between these two corpus-level representations.
Simmvggmbb (Cuains Cuat = cos sim (((C@an e((Cws))
Average Max Semantic Similarity: Quantifies how well each test sample represented in the training set. This metric is explicitly an aggregation of sample-level scores. For each test sample zest we find its highest cosine similarity to any sample in the training set, and then average these maximum similarity scores.
Sim AvgMax
max Przan ECizin
cos cos_sim(emb(Area)) emb(Taain))
Crest
Tian ECent
Average Pattern Conformity: Assesses how well test samples align with the dominant semantic patterns of the training set. We first run k-Means on the training embeddings to find & centroids =- The metric is the average of each test sample's maximum cosine similarity to any of these centroids, making it clear aggregation of sample-level conformity scores.
PatternConformity
Σ
max cos cos_sim(emb(Fees) ct) iie (I....)))
Crest
Frass ECros
B.6 VERIFICATION OF CONTROLLED DISTRIBUTIONAL GAPS
To verify that our stratified splitting method effectively created controlled distributional gaps, we visualized the similarity between each training data level and the test set. Figure S shows the distribution of cosine similarities, confirming a systematic shift where Level most similar the test set and Level 5 is least similar. decrease across five different lexical and semantic similarity metrics, validating the integrity of our experimental setup.
corroborates this by demonstrating monotonic
emotion: troin similarty bins
rotten tomatocs: train similanty bins
0.6
0.7
0.9
0.4
0.4
0.-
0.9
Bath susset Jower
coser centrod)
Trais subset Uoweer lovel cloer test controci
(a) emation dataset
(b) rotten_tomatoes dataset
Figure 5: Similarity bins for the emotion (left) and rotten_tomatoes (right) datasets. Both plots show the distribution of cosine similarities between training samples in each level and the test set centroid, confirming that Level is most similar and Level 5 is least similar. It is worth noting that the levels are not perfectly discrete, which is natural consequence of our stratified splitting strategy. samples are partitioned within each class before being aggregated into global levels.
16

---

Published as a conference paper at ICLR 2026
Emotion: Micro Nacro Nutrics
aim
E--EF Cullte
sroudd
5.000
0.2
whe Nin
5.90
044
5.85
Lea
Dis Lovll
Das Lond
Rotten Tomatoes: Wirro vs. Mocro Netrfccs
leroar Smelariy
0.50
B. Orann
Aes Ersceding
Aug Mie Semarik Sim
Art Fatrer Codienlty
SomeerTer Nacss
3..99
0.990
0352
548
Figure 6: Verification of controlled distributional gaps using five similarity metrics for the emotion (top panel) and rotten_tomer tomatoes (bottom panel) datasets. Both panels show a consistent decline across all metrics as the level increases from 1 to 5, validating the successful creation of a widening train-test gap.
C DETAILED EVALUATION STATISTICS AND SANITY CHECKS
To further validate the robustness of our main findings, conducted series of auxiliary experiments. These checks were designed to test our conclusions against alternative methodological choices and potential confounding factors, ensuring that the observed decontamination effect of knowledge distillation is genuine and reliable phenomenon. This appendix details the methodology and results of these investigations.
C.1 FULL PERFORMANCE DATA FOR BASELINE, TEACHER, AND STUDENT MODELS
Benehmark
Titen
Tairto
A.ooy
95% CI
Bolesn
Party
Adeeg
95% CI
Shrry
A
95% CI
ZUNowsgroops
70.37
84.62
+1
[13.4, 15.1]
69.38
81.29
+11.91**
11.71.1.7]
68
70.02
+1.12
[0.9.20]
91.89
99.25
17.30
[6.8. 8.0
91.64
98.31
16.67
[6.1,7.3
92.24
10.65'
[0.3, 1.0]
Banking77
78.65
83,60
+4.95 ...
13.7,
76.97
82.29
...
[4.1.66]
62.21
64.44
Emotion
92.29
98.35
16:08
[.10.
92.70
97.59
14.89
[3.9,5.9
93.10
93.76
10.66
[-0.0. 1.4)
IMDs
93.63
99.81
+8.18
6.59
92.78
99.65
+8.87
[6.6. 7.21
92.72
93.39
+0.67
[0.5, 0.91
Tomatoes
85.14
98.65
*
[11.5, 15.6]
84.20
97.52
+13.52
15.4)
83.77
84.33
+0.58
[-0., 1.6)
SNLI
86,00
98.72
*
[12.1, 13:4]
82.95
96.10
+13.14"*
[12.5, 13.8)
52.91
58.67
+5.77
(1.7
Tweet Sentiment
67.81
96.82
+29.01
[28.2,
29
67.53
92.99
425.66
[34.9,26
66.99
70.24
+3.25
[2.7. 3.8
Table 8: Detailed performance statistics using notation defined in Table 4
Values are percentage points. AAce denotes the absolute performance gain from contamination (Dirty Clean). Significance ("p < 0.05,* ** < 0.01, *** < 0.001) and 95% Confidence Intervals (CI) are reported.
Table g presents the complete dataset corresponding to the figure in the main body that illustrates the performance of the baseline, teacher, and student models. This table provides detailed breakdown of the results.
C.2 ROBUSTNESS ACROSS DISTILLATION STRATEGIES: THE PRIVACY-UTILA TRADE-OFF
To address the question of whether specific distillation objectives facilitate or hinder data laundering, we extended our analysis include Soft Forward, Soft Reverse, and Hard-label distillation, comparing "Pure" (teacher-only) signals against "Mixed" objectives (teacher + ground truth). The results are detailed in Tables 9 10] and
While our primary conclusion holds that distillation generally acts as an effective decontaminator a closer inspection reveals how different distillation strategies influence the risk of data laundering:
17

---

Published as a conference paper at ICLR 2026
Benchmark
Sooft.fud clean
Sseft.fud Siirty
AAoe
95% CI
sod.t.fdd..ia cleam
sooft.fud.miz dorty
AAces
95% CI
20Newsgroups
68.60
70.02
+1.42"
[0.9.
69.04
70.13
+1.09
[0.7. 1.5]
AG News
91.59
92.24
+0.65"
[0.3, 1.0]
91.62
92.26
+0.64
[0.3. 1.0]
Banking77
62.21
64.44
+2.23
[0.6, 3.9]
77,91
78.28
[-0.4,
Emotion
93.10
93.76
+0.66
[-0.0, 1.4]
92.86
93.17
+0.31
[-0.2, 0.8]
IMDb
92.72
93.39
+0.67**
[0.5.
92.71
93.76
+1.05
[0.9. 1.3)
Rotten Tomatoes
83.77
84.33
+0.56
[-0.4, 1.6]
84.07
84.65
+0.58
[-0.2, 1.4]
52.91
58.67
+5.77
[4.7.
68.52
74.18
[4.8, 6.51
Tweet Sentiment
66.99
70.24
+3.25
[..7.3.8]
67.20
69.63
12433
[2.0, 2.9]
Table 9: Comparison of Student accuracy: Soft Forward (Pure) vs. Soft Forward (Mixed)
distillation strategies.
Benchmark
clean
sdoft.reu dirty
AAccs
95%
sdoft.reu,ms Clean
ssoftrrev,mts dirty
AAces
95% CI
20Newsgroups
68.90
70.20
+1.30
[0.7, 1.9]
69.16
70.19
+1.03**
[0.6, 1.4]
News
91.47
92.09
+0.63"
[0.2, L.]]
91.56
92.26
+0.70"
[0.4,
Banking77
61.68
63.34
+1.66
[0.0, 3.3]
78.68
78.87
+0.19
[-0.5, 0.9]
Emotion
93.11
93.48
+0.37
[-0.2. 1.0]
92.87
93.25
+0.38
[-0.1, 0.9]
IMDb
92.68
93.29
+0.60
[0.4, 0.8]
92.69
93.70
+1.01
[0.8, 1.2]
Rollen Tomatoes
83.71
84.09
+0.38
[-0.6, 1.4]
84.15
84.28
+0.13
[-0.6, 0.9]
SNLI
52.84
58.54
+5.70
[4.7, 6.71
65,84
71.76
+5.91
6.88
Tweet Sentiment
67.29
70.20
+2.92***
[2.4.3.5]
67.28
69.77
+2.49
[2.0,
Table 10: Comparison of Student accuracy: Soft Reverse (Pure) vs. Soft Reverse (Mixed) distillation
strategies.
Benchmark
Shard
dirty
AAcos
95% CI
chard,mia
clean
hhardmmix derrty
AAccs
95% CI
20Newsgroups
69.10
69.57
+0.48
[-0.0, 1.0]
69.27
70.18
+0.90
[0.4, 1.4]
AG News
91.72
91.80
+0.08
[-0.3, 0.5]
91.66
92.16
+0.49*
10.1,
Banking77
66.61
66.42
-0.19
[-144, 1.0]
70.87
71.00
+0.13
[-1.1, 1.4]
Emotion
92.88
93.57
+0.69
[-0.0, 1.51
92.90
93.34
+0.44
[-0.3, 1.21
IMDb
92.79
93,06
+0.27
[0.0, 0.5]
92.82
93.58
+0.76***
[0.6, 1.0]
Rotten Tomatoes
84.15
84.11
-0.04
[-1.0,0.9]
84.18
84,45
+0.26
[-0.8, 1.3]
SNLI
52.69
58.17
+5.47
[4.4, 6.5]
69.36
74.23
+4.87
5.70,
Tweet Sentiment
67.16
69.33
+2.17
[1.6, 2.71
67.26
69.01
*
11.3, 2.31
Table 11: Comparison of Student accuracy: Hard Label (Pure) vs. Hard Label (Mixed) distillation
strategies.
18

---

Published as a conference paper at ICLR 2026
1. Soft targets earry more risk than hard labels. Across benchmarks, Student models trained on soft labels consistently exhibit larger performance gaps (AAccs) compared to hard labels. This vul- nerability is reflected in statistical significance. While Hard Pure distillation often yields insignificant results (indicating that no systematic laundering occurred), Soft methods frequently exhibit highly significant gaps (**,, p < 0.001). We attribute this to the information richness of soft logits, which carry the teacher's full probability distribution, including the patterns learned from contamination. In contrast, Hard labels act as filter, removing these subtle nuances and preventing systematic laundering.
2. Mixed objectives can increase the visibility of data laundering. The "Mixed" strategy, which combines the teacher's signal with the training data's s ground-truth labels, tends to widen the perfor- mance gap (AAcog). This effect is most pronounced in hard distillation (Table IT). where adding the ground truth often makes the laundering effect statistically significant. We hypothesize that the fround-truth signals may synergize with the signals from the contaminated teacher, making the laundering effect more pronounced in the student compared to when learning from the teacher alone. However, the exact mechanism behind this interaction requires further investigation.
3. An Open Question: The Privacy-Utiite Trade-off. These findings highlight a dilemma rather than simple solution. On one hand, using Pure Hard Distillation appears to be the safest method for decontamination, as minimizes the transfer of test-set patterns. On the other hand, Soft Mixed Distillation provides significantly higher utility (e.g., Banking77 clean student accuracy increases from ~ 66% in Hard Pure to 78% in Soft Mixed). Limiting the student to sparse information (Hard labels only) hampers learning efficiency. Therefore, how to balance the need for rich training signals against the risk of data laundering remains an open question for future research.
C.3 GENERALIZATION ACROSS ARCHITECTURES USING STATE-OF-THE-AN LLM TEACHERS
To address the potential limitation of relying solely on BERT-based models and to validate the universality of our findings, we extended our analysis include state-of-the-ar lightweight Large Language Models (LLMs) as teachers. Specifically, we employed Llama3.2- and Qwen3-0.6B. These models represent the current saaee--f-the-aaee for their size class.
We maintained distilbert tas---bass--aa-na uncased as the fixed student model to strictly isolate the influence of the teacher's architecture and capabilities. All experiments in this section utilize the Soft Forward distillation strategy (a = 1, r = 2.0). Table T2 presents the detailed results, including 95% bootstrap confidence intervals (CI).
The evidence strongly supports our central claims. Despite employing highly capable teachers that memorize contaminated data extremely well, including perfect accuracy on Tweet Sentiment and Rotten Tomatoes, the behavior remains consistent. The laundering effect that reaches the student is still minimal. For instance, on Tweet Sentiment, while the Llama-3.2 teacher gains 25.80% from direct contamination, the student's laundering gain is limited to just 1.91 %. The confidence intervals indicate that the laundering effect is statistically significant only for the Tweet Sentiment benchmark (its intervals excludes zero). Importantly, even in this significant case, the estimated magnitude is still drastically reduced compared to the direct contamination gain. This pattern reinforces the role of Knowledge Distillation buffer to prevent the propagation of contamination from the teacher model.
C.4 PRETRAINING CORPUS OVERLAP AUDIT
potential concern is that the base models used in our study, BERT and DistilBERT, might have been inadvertently exposed to benchmark test data in their original pretraining corpora. To ensure that our findings originate from our explicit contamination protocols rather than from such pre-existing issues, we conducted corpus-level overlap audit. According their respective documentation. both BERT and DiRt were pretrained on a combination of English Wikipedia and the BookCorpus (Zhu et al. 2015). We created surrogate pretraining corpus composed of recent snapshot of Wikipedia
19

---

Published as a conference paper at ICLR 2026
Model
Benchmark
Totcon
Tairey
Doteon
Datvey
A.ooe
95% CI
"dlenn
Sdirty
A.oog
95% CI
Agnews
92.58
99.98
91.09
98. 32
+-7.3
[6.6
91.08
91,46
+0.39
-0.01, 0.78
Emotion
92.81
99.80
92.90
97 69
+4.79
[399,
92.81
95.11
+0.30
-0.21, 0.82
LLAMA3.2-1B
Rotien Tomatoes
90.66
100.00
84.11
97. 56
13.45
[11.4, 15.6
84.15
84.32
10.17
0.62, 0.98
Tweet
100.00
93.08
4-25.80'1
25.0,26.0)
+1.91
[1335,
Agnews
99.98
98.32
[6.7..88
91.04
(0.08,0.22)
Emotion
92.71
99.75
444.9 ***
[3.8,688
92.82
93.12
0.25, 0.86
QWEN3-0.6B
Rotten Tomatoes
87.15
100.00
84.11
97.56
13.45
15.5
84.22
84.37
+0.15
0.73, 103
Tweet Sentiment
68.49
99.99
67.28
93.08
25.80
(2.00, 26.6)
67.10
69.10
12.00
[1,43,
Table 12: Data Laundering results using modern LLMs (LLAMA3.2-1B and QWEN3-0.6B) as teachers. The Student model is fixed DistilBERT trained with Soft Forward distillation. Gain denotes the absolute percentage point increase. Significance (***) indicates < 0.001. 95% Confidence Intervals (CI) are computed via bootstrapping.
('20231101er) 1en.enn and the BookCorpusOper Subsequently, performed an exhaustive search to determine if any sentence from the test sets of our eight benchmarks appeared verbatim within this extensive corpus.
The audit revealed zero exact 13- 13-gram matches between any of our benchmark test sets and the surrogate pretraining data. While this does not preclude more subtle forms of semantic overlap, it provides strong evidence that our results are not confounded by the most direct form of test set leakage into the pretraining pipeline of the models we used. This finding inereases our confidence that the leakage effects studied in this paper are indeed a consequence of our controlled experiments.
Phtrss//huggingffcc.cm hhtt:://whnnigggaac.ooossssswwwr cwdddsstwiiikiiiiviiiiiiiii
20


Reviews:
[{"review_id": "xEATEnyb3g", "reviewer": "Reviewer_7UAR", "summary": "This paper investigates whether knowledge distillation (KD) from a contaminated teacher to a clean student actually “launders” test-set knowledge, thereby inflating evaluation results even when the student never directly sees test data. Using eight text classification/NLI benchmarks, the authors compare four families of models per dataset: (i) clean vs. contaminated baselines trained directly on data; (ii) clean vs. contaminated teachers; and (iii) students distilled from clean vs. from contaminated teachers, always on clean data. They define benchmark-level leakage ($\\Delta$ Acc) and sample-level leakage ($\\Delta$ laund vs. $\\Delta$ contam) to separate direct contamination from laundering and show three main things:\n\n1. direct contamination is large and highly significant across all tasks;\n\n2. laundering exists but is much smaller and often statistically insignificant;\n\n3. laundering and contamination are only weakly correlated at the sample level, so laundering is not just a \"weaker\" version of contamination.\n\nThe authors further hypothesize and confirm via controlled experiments on emotion and rotten_tomatoes, that laundering becomes more visible when the train-test distributional gap is deliberately widened, i.e. when test data is semantically farther from training data. The conclusion is cautiously positive: KD can serve as a decontamination buffer in many realistic settings, though it is not perfect and can fail on benchmarks with large train-test shifts.", "questions": "1. Low-contamination regime: What happens if the teacher sees only 5–10% of the test set (or paraphrased test items) during fine-tuning? Does the KD bottleneck still suppress laundering to sub-1%?\n\n2. Other teacher-student pairs: Would the same conclusion hold for (T5, decoder style) where the capacity gap is bigger and the teacher is stronger? Right now we don’t know if stronger teachers launder more.\n\n3. Generative settings: Can the authors comment on summarization or code evaluation benchmarks, where \"exact match\" is less relevant but pattern transfer is easier?\n\n4. Realistic leakage patterns: Your contamination is \"inject full test + remove train\". Have you tried something like \"mix 10% test into train without removal\", i.e. inflate total size, which is closer to pretraining-style accidental inclusion?\n\n5. Metric sensitivity: On the datasets where laundering was \"not statistically significant\", what were the actual deltas and Cls? A small but systematic 0.5-0.8pt lift is still operationally important for competitive leaderboards.\n\n6. Defense implication: If KD is to be used as a decontamination step, what hyperparameters (α, temperature, loss type) minimize laundering while keeping accuracy? A table of “safe KD settings” would make the paper much more useful.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear problem formulation and motivation: The paper isolates a concrete, timely question that sits right at the intersection of model compression and benchmark integrity. \n\n2. Well-controlled experimental design: The two-stage setup (dirty vs. clean teacher to distilled student trained only on clean data) is clean and makes causal interpretation easier: any gain of the dirty student over the clean student must have come through the teacher. The inclusion of clean/dirty baselines with the same student architecture is a good control. \n\n3. Breadth across 8 benchmarks: Using topic, sentiment, intent, emotion, NLI gives the results some external validity, and they correctly note that laundering is benchmark-specific. \n\n4. Sample-level analysis: Moving from aggregate accuracy to per-sample difficulty and leakage scores ($\\Delta$ laund, $\\Delta$ contam) is a real contribution-it shows weak Pearson correlations and even slightly negative for SNLI, supporting the \"distinct mechanisms\" claim. \n\n5. Distribution-gap experiment: The controlled 5-level stratification that gradually pushes training data away from the test distribution is thoughtful and directly tests their hypothesis; it’s rare to see this done for contamination work. \n\n6. Takeaway for benchmark designers: The paper ends with an actionable message. Multiple test sets at different distances reduce the risk of KD-based laundering. It's not just diagnosis but also design guidance.", "weaknesses": "1. Limited architecture and task scope: All core results are on BERT-base to DistilBERT, encoder-only, classification tasks, which the paper itself mentioned this. This significantly limits the generalizability of the findings. The conclusions from this paper may be spurious and may not extend to other conditions (task, architecture, scale). Additionally, the original \"data laundering\" issue is strongest for decoder-only LLMs and generative or instruction-following tasks, where KD often uses richer signals. The current evidence base is therefore narrower than the title (\"revisiting the concern\") suggests. A reviewer can accept the current claim only as: \"for small/medium English classification with BERT-like models, KD is mostly a buffer\". Generalizing to LLMs is still open. \n\n2. Contamination protocol is single-style and generous. They inject the full test set and remove an equal number of training samples. That's an extreme, clean, known contamination pattern; real-world pretraining leakage is usually partial, fuzzy, or paraphrased. It’s not obvious that a KD bottleneck will stay this effective under messier or low-rate contamination. The paper should at least discuss low-contamination regimes or partial overlaps. \n\n3. Effect sizes sometimes tiny, but emphasized strongly. On several datasets the student’s dirty–clean gap is <=1% and sometimes statistically non-significant. The paper’s narrative (\"KD as decontamination\") leans a bit harder than the numbers strictly justify; one could also say \"we failed to induce consistent laundering except under adversarially widened gaps\". A more cautious framing would help. \n\n4. Train-test gap story is suggestive, not airtight. They do show that laundering becomes more detectable as the gap widens, but on emotion the magnitude itself doesn't clearly grow-only the p-values get smaller. That weakens the causal reading (\"gap -> laundering magnitude\"), at least for one dataset. The paper should disentangle: (i) teacher now actually learns test-specific patterns vs. (ii) student variance shrinks so we can just see the effect. \n\n5. No comparison to other KD objectives meant for privacy/decontamination. They mention reverse KL and mixed-objective settings in the appendix, but the main narrative doesn't tell us whether changing the distillation loss makes laundering harder or easier. That would be highly actionable. \n\n6. Evaluation mostly on accuracy. Since laundering is about \"benchmark integrity\", one might want to know whether rank-based, calibration-based, or subset metrics are more vulnerable. If laundering only moves 0.6-1.4 accuracy points, is that actually a practical threat, e.g. to leaderboard positions? The paper doesn’t quantify that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZrYWF2a7Od", "reviewer": "Reviewer_Uoo6", "summary": "This paper systemically revisits the problem of data laundering—the unconscious transfer of test-set knowledge via knowledge distillation (KD) from a contaminated teacher model to a clean student. \n\nFirst, this paper conduct systematic experiments across eight text classification benchmarks using BERT-based teachers and DistilBERT students. They find that performance gains from laundering are significantly smaller and often statistically insignificant compared to those from direct contamination, which suggests that KD may indeed function as an effective decontamination method.   \nSecondly, through sample-level correlation analysis, this paper shows that laundering is a distinct mechanism rather than a diluted form of contamination.   \nFinally, this paper demonstrates that laundering effects intensify when the train–test distributional gap widens, suggesting benchmark-specific susceptibility rather than universal vulnerability.", "questions": "1. Given the train-test domain gap finding in this paper, can the author provide some guidance or suggestions for benchmark curation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. This paper provides the first large-scale, controlled study that quantifies the prevalence and mechanisms of data laundering. Compared to the pioneer work (Mansurov et al. 2025), the experimental setting in this paper is more scientific and comprehensive, which provides more convinced understanding for data laundering problem in KD. \n\n2. This paper suggests that KD actually generally reduces contamination rather than propagating it, which mitigates the concern of its usage in practical, especially in the LLM era.", "weaknesses": "1. This paper mainly explores BERT-based models, which is kind of out-of-date in the LLM era. It would be better that if this paper could explores the data laundering issue in GPT-style models in future works, as it's more popular and practical recently. \n\n2. Based on thorough experiments in this paper, it would be better if this paper can provides some guidance for test set or benchmark curation in terms of train-test domain gap. I believe this can be a potential application for the data laundering issue in practical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BeK3VNYzPS", "reviewer": "Reviewer_Vwqr", "summary": "This paper examines whether knowledge distillation (KD) can transfer test-set leakage from a contaminated teacher to a clean student. Across eight benchmarks, the study finds that while data laundering exists, its impact is generally small and far weaker than direct contamination. The results suggest that KD largely mitigates test-set leakage rather than amplifying it, indicating that KD can serve as an effective decontamination technique in most cases.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an interesting question: the role of KD in data laundering.\n- The authors conduct a large-scale assessment across eight benchmarks to determine the prevalence and magnitude of data laundering. \n- The authors identify that the effect of data laundering via KD is substantially smaller than that of direct contamination.", "weaknesses": "- The clarity and readability of the paper would benefit from a more explicit and formal definition of direct contamination in the introduction.\n- The study focuses solely on encoder-based BERT and classification task. Given the prominence of decoder-only LLMs and the rising importance of generative tasks, where KD and contamination could be more complex and potentially more harmful, the generality of the current findings to modern LLMs is unclear. Furthermore, the applicability of the conclusions to other modalities such as images also remains unaddressed.\n- The authors' experiments assume KD is performed strictly using clean ID data. However, in generator-based data-free KD settings (e.g., [1–2]) or KD pipelines for generative models (e.g., [3]), if a teacher is contaminated, the generated synthetic data itself may embed and transfer leaked knowledge, making laundering more pronounced. The authors should at least discuss the applicability and limitations of their conclusions under such settings, where leakage pathways may differ from those in standard supervised KD and classification tasks.\n- Some references are incorrect: \n\"Mingyuan Hong, Yicheng Li, Zijian Yang, Yisen Wang, Zhangyang Wang, et al. Revisiting data-free knowledge distillation with poisoned teachers.\" the first author should be Junyuan Hong.\n\n[1] Revisiting data-free knowledge distillation with poisoned teachers. ICML 2023\\\n[2] When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need. ICML 2025\\\n[3] A Survey on Knowledge Distillation of Large Language Models. ArXiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "liwyyoEccz", "reviewer": "Reviewer_TbMs", "summary": "This paper investigates Data Laundering. It transfers test-set knowledge from a contaminated teacher to a clean student via distillation. Through a systematic study on eight classification benchmarks, the authors find these concerns are largely overstated. They demonstrate that performance inflation from laundering is minimal compared to direct contamination and often statistically insignificant. They identify the train-test distributional gap as the key driver for the rare instances of laundering. The paper argues that knowledge distillation, rather than being a liability, acts as an effective \"decontamination\" technique, robustly mitigating test-data leakage.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Timely and Significant Topic:** As concerns about data contamination in large pre-training corpora grow, understanding the downstream effects of \"laundered\" knowledge through common techniques like distillation is critical for the ML community. This paper provides a data-driven analysis of a widely held concern.\n\n2. **Empirical Rigor:** The study systematically isolates the laundering effect from direct contamination across eight diverse benchmarks. The use of bootstrapping for statistical significance and the robustness checks across different distillation strategies (soft/hard labels, mixed/pure loss) strengthen the credibility of the findings.\n\n3. **Novel Insights:** The paper's most valuable contribution is not just in quantifying the laundering effect but in identifying when it occurs. The hypothesis and subsequent controlled experiments that link laundering to the train-test distributional gap are a novel and important finding.", "weaknesses": "1. **Scope of Models:** The experiments are confined to BERT-base models. While this is a controlled and reasonable setup, the community's primary contamination concerns now center on massive-scale decoding LLMs.\n\n2. **Scope of Tasks:** The study is focused exclusively on classification. Data laundering in generative tasks (for exmaple,  text generation) is a more complex and arguably more insidious problem, as the \"knowledge\" being transferred is far richer than class logits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)