Detecting Data Contamination in LLMs via In-Context Learning
Authors: Michał Zawalski
Meriem Boubdir
Klaudia Bałazy
Besmira Nushi
Pablo Ribalta
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Contamination
In-context learning
Primary area: datasets and benchmarks
TL;DR: We propose Contamination Detection via Context (CoDeC), a simple, efficient, and model-agnostic method that detects training data contamination in LLMs by measuring how in‑context examples affect predictions.

Abstract: We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in‑context examples typically boost confidence for unseen datasets but may reduce it when the dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and dataset-agnostic, making it easy to integrate with benchmark evaluations.

Published as conference paper at ICLR 2026
DETECTING DATA CONTAMINATION IN LLMs VIA
IN-CONTEXT LEARNING
Michal Zawalski
Meriem Boubdir NVIDIA mboubdir@@vicia..
Klaudia Balazy NVIDIA
NVIDIA
Besmira Nushi
Pablo Ribalta
NVIDIA
NVIDIA
priba wtw.wiiiiidiiacmm
ABSTRACT
We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in-context examples typically boost confidence for unseen datasets but may reduce it when the dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and Cataset-agnostic making easy to integrate with benchmark evaluations.
1
INTRODUCTION
Detecting contamination is essential for the integrity of LLM evaluation (Dodge et al., 2021 Maini et al.
2024 Carlini al.) 202T Singh et al., 2024). Existing approaches, such as loss-based criteria (Shi et
2024 Zhang et al. 2024 Carlini et al. 2021), score calibration using external models (Maini al. 2024
Carlini et al. 2021), and explicit overlap checks (Gao et al. 202T Biderman et al., 2023). often require access training data, extensive parameter tuning, and may fail produce reliable, interpretable results for large models. This calls for an automated, scalable method applicable across diverse datasets and architectures.
To fill this gap, we introduce Contamination Detection via Context (CoDeC), simple yet effective datasct-level method that detects data contamination via in-context learning. CoDeC builds on the observation that context sampled from dataset seen during training provides
information the model, while unseen data improve predictions as in few-shot learning (see Figure T]. Hence, the percentage of samples in a dataset for which adding context from the same dataset reduces confidence quantifies contamination. CoDeC requires only gray-box access to token probabilities, works with any dataset, and needs no prior knowledge of the model's training corpus.
Unlike traditional methods focused solely on strict membership inference, CoDeC detects contamination arising also from training on augmented or related data, such synthetically generated shadow data, which
Work done while at NVIDIA.
"The code run CoDeC can httpa//aaaaaaattet......aaaaaaaa
pa packages httaa//eeeeeeceeeaaaaaao/o////e/
eva httpa//www.aaaaaa//ooooooooaaoaa
.pynb

---

Published as conference paper at ICLR 2026
LOBTON VINOUE COTNR
(a) Changes in model confidence unseen datasct.
(b) Changes in model confidence training dataset.
Figure Change model pog-bbbbaiilities with and without context for the Pythia 1.4B model, evaluated on a sample from unseen dataset (gsm8k, [Tab and training dataset (ArXiv, lb). Differences in confidence for consecutive tokens consistent. See Appendix C.2 Tor details of this experiment.
exposes the model to distributionn-peccficc cues. By capturing both direct memorization and contamination via related distributions, CoDeC proves valuable both for fair evaluation and guiding model development.
Through extensive experiments, we show that CoDeC clearly separates training from unseen datasets, achieving near-perfect dataset-level AUC 99.9% across many models, while baseline methods fail. We apply CoDeC recent open-source models, evaluating on widely used benchmarks. finding multiple instances of potential contamination. Our ablations further demonstrate that CoDeC is stable across diverse datasets and model types, and can be further enhanced by scaling the context. CoDeC enables the community to better trust reported LLM results and supports the development of more reliable model evaluation practices.
In summary, our main contributions are the following:
We introduce CoDeC, a simple model-agnostic approach that effectively detects data contamination in LLMs using in-context learning.
We confirm the strong advantage of CoDeC over baselines through experiments on diverse models and datasets, analyze its properties, and propose practical guidelines for applying it.
We use CoDeC to evaluate variety of recent models on widely used benchmarks, revealing several instances of potential contamination.
2 CONTAMINATION DETECTION VIA CONTEXT
2.1 PROBLEM DEFINITION
Given language model . and candidate dataset D = {z; J where each x; is text sequence, goal to quantify contamination, i.e. whether that dataset or similar data was the training set and the model is relying memorization rather than generalization.
2.2
KEY IDEA
LLMs respond differently in-context examples depending prior exposure. When given unseen dataset, adding in-context samples taken from that dataset usually improves the model's confidence, as it can generalize better with more information (Brown et al.) 2020). However, if the model has memorized the dataset, the in-context samples not only provide little new information but also disrupt memorization patterns, leading to reduced confidence (Razeghi al. 2022).
Thus, by comparing confidence levels with and without in-context learning across sample sequences, we can leverage these shifts to detect contamination. Note that generation is involved, measure only logits
2

---

Published as conference paper at ICLR 2026
Fix target datapoint
Sample more datapoints
Evaluate the suspect LLM
change in logits N
Calculate the score percentage
Suspect dataset
Score (D)
LLM
[ <0
Bandemm context Target samples
contest samples
sample
Figure 2: Overview of Contamination Detection via Context (CoDeC). For each dataset element, CoDeC augments the context with a few other samples from the same dataset. A decrease in the model's logits for the target sample indicates potential contamination. The overall contamination level is estimated as the fraction of samples exhibiting this effect.
of the given sequences estimate confidence. We provide experimental evidence for those observations in Section and Appendix C.2 and extended discussion Appendix A
2.3 CoDeC PIPELINE
Our method (presented in Figure 2 consists of the following steps:
1. Baseline prediction:
For each datapoint x: in the suspect dataset D, obtain the model's average log- likelihood on the consecutive tokens of æ.
2. In-context prediction: Sample n additional examples 21, Fn from D {x} prepend them to x (creating
a sequence x [..|/=,,, [r), and obtain the model's predictions on this new context.
3. Score computation: Compute the difference in confidence Axx) logprobm..come (2r) logprob (1)).
4. Aggregation: Repeat the above for all D. The contamination score for the dataset then
SCoDeC (D)
(a;))) 0]
N
where 1. is the indicator function.
2.4 PROPERTIES
CoDeC offers several properties that make it broadly applicable and easy to use. It produces intu- itive, percentage-based contamination scores that are immediately interpretable and require no model- or dataset-specific calibration, avoiding the threshold tuning often needed in membership inference methods
due to the arbitrary scales their raw scores (Maini et al. 2024 Carlini et al. 2021 Shi et al. 2024).
The method works with any dataset that can be represented as collection of text sequences, including model-agnostic and requires only gray-box access
standard training corpora and widely used benchmarks. to token probabilities. CoDeC is also computationally efficient, needing just two forward passes per sample. A detailed discussion of its theoretical and empirical propertics can be found in Appendix
2.5
WHY DOES CODEC WORK?
The effectiveness of CoDeC stems from several intuitive principles:
Dataset-specific priors reveal contamination. Models trained dataset internalize its unique style, structure, vocabulary, and implicit assumptions. these priors are already memorized, additional co-textext examples add little useful information.
3

---

Published as conference paper at ICLR 2026
2. Context disrupts memorization. For contaminated datasets, adding memorized in-context examples can interfere with memorized token sequences. The presence of patterns triggering memorization in the context confuses the model, leading reduced confidence.
3. ICL simulates finetuning dynamics. Contaminated models resemble finetuned models near saturation additional training yields minimal gains, whereas non-contaminated models adapt faster. CoDeC effectively measures how much learning capacity remains for the dataset. Figure g iillustrates these dynamics.
4. Interventions expose loss landscape. Contamination is often linked overfitting (Maini et al., 2024). Overfitted models sit in narrow local minima that are more easily destabilized by new context, while unseen data corresponds to flatter, more stable loss surfaces.
the next section, we provide empirical evidence grounding those intuitions. Further discussion can be found in Appendix A.5
3 EXPERIMENTS
In this section, we demonstrate that CoDeC consistently distinguishes between seen and unseen data, is stable across evaluation settings, and yields interpretable scores for model auditing.
Models. We validate CoDeC diverse suite of LLMs with publicly available weights and training data,
enabling reproducibility and ground-truth verification. Our evaluation includes models trained on different corpora and spanning varied architectures: Pythia (Biderman et al.) (023), GPT-Neo (Black al. 2021)
and RWK V-4 (Peng et al. 2023), all trained on the Pile (Gao et al. (021); OLMo (Groeneveld et al., 2024).
trained on Dolma (Soldaini al. 2024);; Nemotron-v2 (NVIDIA al. (025a) and Nemotron-H (NVIDIA
ct al. 20256). trained on Nemotron-CC CC (Su et al. 2024).
Those models cover transformers, RNNs, and hybrid
mamba-transformer architectures.
Datasets. For each model, we create test bed consisting of (1) data known to be in its training set (c.g., subsets of the Pile) and (2) unseen data pub- lished after the model's training cutoff. The unseen data includes recent benchmarks, news, websites, etc. A complete list of the datasets can be found in Appendix
Model
CoDeC (ours)
Vanilla loss
Min K%
Zlih
Pythia 410M
100.0%
75.0%
76.2%
92.3%
Pythia
100.0%
79.2%
Pythia 12B
100.0%
76.9%
82.3%
92.3%
GPT-Neo 133
100.0%
77.3%
79.2%
90.8%
GPT-Neo 20B
100.0%
76.9%
83.5%
92.7%
RWKV 430M
100.0%
75.4%
75.4%
92.3%
RWKV-4
100.0%
76.5%
79.6%
91. 9%
RWKV 14B
99.6%
77.3%
81.5%
92.7%
OLMo
100.0%
71.9%
Baselines. We compare CoDeC against three com- mon contamination detection methods: Vanilla Loss (Fu et al., (0244) scoring based on model loss; Min-K% (Shi et al.) (024). focusing the most informative tokens; and Zlib Ratio (Carlini et al. 2021). which normalizes perplexity by sample entropy. More details can found in Appendix C.I
OLMo 7B
100.0%
65.6%
72.7%
78. 11%
Nemotron v2 9B
100.0%
83.7%
97.6%
98 2%
Nemotron H 8B
100.0%
80.0%
73.3%
88 .P%
Nemotron-H 56B
100.0%
82.2%
86.7%
92.0%
Cumulative
Table 1: AUC scores for separating unseen datasets (Figure computed per datase
3.1 MAIN VALIDATION
CoDeC cleanly separates seen from unseen data (see Figure 3 achieving dataset-level AUC of 99.9% across all evaluated models (see Table M. In contrast, baselines fail to provide a reliable separation. Their scores for
While AUC, parameter-free metric commonly used evaluating MIA methods MMaini et al. 2024). usually computed over individual samples, here it computed dataset-level scores. Details provided Appendix B.4
4

---

Published as a conference paper at ICLR 2026
Urseen
Venilla loss baseline
Min-K% baseline
minatiiination
3
-10
CoDeC (ours)
Zlib Ratio baseline
Contain teose
100%
20%
-10%
Pyttii 410M
Pytia 4888
Pynsa 128
GPr-seo GPT-Hee 208
minsy-4 430M
popre- 148
Hemetron 368 Cumulative
Pythia 410M
Pythis 1.48
Pyttia 128
GpT 1.38
GPT meoo 208
mwsv-4 430M
Awev 168
Nematren 12 98
Hemotron- Remairon-tt 368 Cumulative
macor4
OLA6 GLMD
Nemotron 12
Remotren-H
mway-4
GLMo
GLMo
Figure 3: CoDeC vs. baselines; Contamination scores for training (red) and unseen (blue) datasets. Each point is a model-dataset pair. CoDeC achieves the best separation, enabling consistent classification across models and datasets.
seen and unseen data overlap significantly, making them ineffective for confident contamination detection. The consistent and sharp separation achieved by CoDeC enables meaningful, model-agnostic comparisons.
To further validate CoDeC, we examined how scores for each dataset vary across different models. Figure shows the distribution of CoDeC scores on various datasets for all models trained on the Pile corpus (extended results be found in Appendix C.3) For most datasets scores remarkably consistent, clustering around the -ptaset-specific mean located between 0 and 60%. Such stability suggests that major outliers in the distribution are indicators of at least partial contamination.
We also analyzed whether dataset diversity affects CoDeC scores by measuring the variance of text embeddings (see Figure 5). While a certain correlation exists, it is insufficient to explain the observed patterns, indicating that CoDeC captures deeper signals beyond simple diversity. Importantly, this finding suggests that scores even near 50% should not be treated as contamination hints for highly diverse datasets, especially the scores are consistent across models.
3.2 CONTAMINATION DURING TRAINING
Figure tracks how CoDeC scores change during training of the OLMo 7B model, for which training checkpoints are publicly available. At initialization, scores are close to 50% for all datasets, reflecting random delta-confidence fluctuations in an untrained model. By 1K steps, the model has acquired basic language understanding but has seen too little to memorize anything. Only the Stack-v4 dataset (code snippets and byte-like samples) drops only to 33% at Ik steps because it does not align with general language skills, leaving memorization as the only viable strategy.
The key shift occurs between Ik and 10k steps, which makes just about 2% of the full training. Contamination scores for training datasets rise sharply, while scores for unseen datasets settle near their final values. Beyond this point, up to the final 477k training steps, scores remain largely stable indicating that the model has already seen enough data to recognize and memorize the distributions of its training sets.
Since CoDeC detects contamination in the very early stages of training, it is highly effective for identifying and preventing benchmark leaks, supporting more reliable model development. The complete list datasets presented in Figure 4 can be found in Appendix C.4
5

---

Published as conference paper at ICLR 2026
CoDeC Scores During Troining
Score Destr but on per Dotooet
Gataset Ders ty
Deteet lags Letten dicwns
CeBec diceptonn
007
CSNSX (lxt)
Sack
Contaminatioe
60%
Hom 30C prodero
SPCL-3
Guterbee Redkico
Hattematio
0,20
Figure 4: Changes in CoDeC scores over training OLMo 7B. A list of the datasets evaluated this experiment can be found Appendix C.4
Figure 5: Distribution CoDeC scores per dataset, computed for models trained on the Pile. The last 5 datasets parts the training data. The full evaluation can he found in Appendix
Contamination during finetuning
Contamination Score
100%
Model
Beuthenkipptt 1-44
20%
Beutherkiggr neo---nt--
Jions 38-1
Queen/Dwen3
20
20
Finetuning batches
Finetuning batches
Finetuning batches
Finetuning batches
Figure 6: CoDeC scores during finetuning different models on 10 datasets (see Appendix B.3 for the list). Though the Qwen3 model does not disclose its training data, by finetuning we can confirm the effectiveness CoDeC also for architecture.
3.3 CONTAMINATING VIA FINETUNING
Finetuned models approach 100% CoDeC score all datasets. further validate CoDeC, we performed a controlled finetuning experiment on four example models (Pythia, GPT-Neo. OLMo, and Qwen3) as shown Figure For each run, took model and finetuned data and from unseen sources. In each case, the CoDeC score consistently rose above 90%, showing a clear and stable increase after exposure the data. Finetuning on a dataset clearly introduces contamination, and CoDeC reliably detects this effect.
chosen dataset, both from its original training
This setup is independent of any uncertainty about the model's original training corpus. As result, the evaluation can be applied to any model, such as Qwen3 (Yang et al. 2025), to confirm the applicability of CoDeC for recent models, even with undisclosed training data.
Contamination transfer. Next, we explore how contamination from one dataset influences related datasets, and vice versa. Specifically, we analyzed the effect of finetuning model on MMLU and measuring CoDeC scores on datasets with varying levels of similarity to MMLU, and conversely, training on related datasets and evaluating contamination on MMLU.
Figure 7 (left) shows how training on MMLU influences CoDeC scores on other datasets. Finetuning makes the model highly contaminated even with unseen questions from MMLU. Contamination scores remain high even when we randomly crop the evaluation data down to as little as 25% of the original content (which was already short).
6

---

Published as conference paper at ICLR 2026
Contarrination transfer to other catasets
Contammination trarefer from other datasets
102%6
120%
Finetuning samples
Contami Cortamination nation Scora
Finetuning test splt
80%
Contaminaainnn
640
GPOA
Random crops (25%)
Auxilarr QA
40%
Gutenberg (OOD)
20%
Finetuning batches
Finztuning batches
Figure 7: Contamination transfer between MMLU and related distributions, computed for the Pythia model. Evaluation other distributions, including data mixes and noise injection can be found Appendix C.5
Model confidence with growing context
Model confidence during finetuning
parill Autaer
Conffidnnce
Confidence
3e-5
1e
0.0
Datisett:
The Pile Itraining)
gamak (unseent
30
40
50
context samples
Finetuning steps
(a) Changes in model contidence with adding more samples to the context.
(b) Changes in model confidence during finetuning with various lcarning rates.
Figure 8: Relation between finetuning and in-context learning for contamination detection. Both plots show how the average probability of target tokens evolve as more learning signal provided. For learning rate around 35 the curves nearly overlap.
Contamination transfers modestly across QA benchmarks because MMLU shares question format, style, and structure with them. Exploiting these patterns can boost benchmark performance without reflecting broader knowledge. Accordingly, models contaminated with QA dataset show slightly elevated CoDeC scores related QA datasets, though far from the high-contamination range. No such transfer observed for unrelated datasets (e.g., MMLU and Gutenberg books).
The right chart in Figure 7
shows the reverse: training on related datasets and evaluating on MMLU. Unsurprisingly, the relative impact is almost identical as in the first case. Even finetuning only parts of the samples or otherwise augmented text drove the CoDeC scores high on the original MMLU, Importantly, this demonstrates that CoDeC cannot be cheated by simple data augmentations.
More evaluations for the contamination transfer experiment, including other augmentations (synthetic rephrases, added noise, whitespace changes, etc.), can be found in Appendix C.5
In-context learning reveals overfitting. We compared finetuning to adding in-context samples and found close relationship between them (see Figure Confidence curves obtained by finetuning with a moderate learning rate and by adding extra context follow identical pattern, suggesting a similar influence model
7

---

Published as a conference paper at ICLR 2026
Example-based Context
Character-based Context
Contamination Score vs. Dataset Size
100%%
100%
Contemination Score
00%
Contaminaton Spore
02%
Oetaset type
Turtr g daidel
Utownn deleals
20%
20%
Dutaset lype
Traniigg darssers
0%
128
517
2048
8199
10
30
1000
300
1000
Number of examples context
Number of characters in context
Dataset Size
(a) Impact of context size.
(b) Impact of the target dataset size.
Figure 9: Ablation studies of CoDeC on the Pythia 1.4B model using 5 training and 5 unseen datasets (see Appendix B3 for the list). Shaded regions show the range between the minimum and maximum scores across 5 runs.
behavior. Contaminated datasets highly sensitive to the learning rate: even small updates cause early drops in confidence of the model. Unseen datasets are robust, with confidence improving even under very high learning rates. This pattern consistent across models and datasets.
These results support the view that contamination is closely related to overfitting (Maini et al. 2024); memorized samples sit in high-vare variability, narrow minima where even small updates can destabilize predictions, whereas unseen samples lie in flatter regions that benefit from additional learning signals. CoDeC captures this distinction, which explains why the method is effective.
3.4 ABLATIONS
Context Size. While our experiments use the simplest form of CoDeC with a single in-context sample, the method can be easily extended to include more context. Larger contexts yield clearer separation between seen and unseen data but also increase computational cost (see FigurePai In practice, a single in-context sample already provides strong signal while keeping inference efficient. However, adding more samples can further enhance performance if resources permit.
Target Dataset Size. A key practical factor is how many examples are needed for reliable contamination scores. CoDeC proves highly sample-efficient:
few 100 examples already give stable estimates with low variance, making the method feasible even on small benchmarks. With 1,000 examples, the variance with respect to the sampled contexts falls below 1% (see Figure (9b). Since the method is statistical, each sample can be viewed as Bernoulli trial to derive confidence bounds easily.
3.5 INTERPRETING CODEC SCORES
CoDeC captures how strongly model's predictions depend on memorized patterns and whether outputs are driven by internal distributional priors. Note that this can also occur if the model has been trained on identically distributed or closely related data (e.g., synthetically generated). Hence, while high scores do not prove that a dataset was in the training corpus, the method is informative for separating generalization capabilities of the model from specialization. Among models with similar benchmark accuracies, models with lower CoDeC scores should generalize better.
For highly contaminated datasets, scores typically cluster near 100%. While most unseen datasets scored below 20%, some can reach as high as 60%. This may occur for two main reasons: (i) the dataset highly related to the training data, resulting in partial contamination; (ii) the dataset consists of extremely
8

---

Published as a conference paper at ICLR 2026
diverse samples, so in-context examples act like random noise. Hence, to separate dataset effects from true memorization, scores should be compared across multiple models on the same dataset.
Best practices. For robust contamination assessment, we suggest combining absolute thresholds with reference-baset, comparisons. High absolute scores (>80%) should be treated as contamination red flags, but significant deviations from other models' CoDeC scores should be equally alarming. For a broader discussion, see Appendix A.5]
3.6 BROADER APPLICATION OF CODDC
Experiments on models with known training data, presented in Section confirm the reliability of CoDeC. With confidence in its validity, subsequently applied CoDeC to broad suite of over 40 recent models and analyzed their contamination scores across widely used benchmarks. The full evaluation can be found in Appendix D
Most models fall well below the conttmmotaiinnionn area, suggesting they were not directly trained on benchmark data. Nevertheless, certain models score close to this threshold remain clear outliers specific benchmarks, which may indicate partial data leakage or substantial training on closely related synthetic derivative content. Variants within the CoDeC's 's robustness across model types. The largest models generally achieve very low contamination scores, supporting the view that increased capacity facilitates generalization over memorization.
model family tend produce similar scores, reinforcing
These findings underscore the support of CoDeC in identifying contamination risk, interpreting benchmark accuracy with appropriate caution, and informing fair model comparison.
4 LIMITATIONS AND FUTURE WORK
While CoDeC demonstrates robust and promising results, there are several areas for improvement. Scores may be affected by adversarial datasets, such as those with heavy duplication or mixtures of unrelated sources, although these issues do not compromise evaluations standard benchmarks (see the discussion Appendix A.3.2]. In principle, may possible to engineer training strategies that bypass increases CoDeC estimates. However, if such approaches preserve model's 's ability to learn and prevent it from internalizing dataset-specific priors, they would represent a notable advance in their own right. Improving the criterion that untrained models unrelated data mixtures consistently yield scores well below 50% could further enhance interpretability. Another direction is adapting CoDeC for strict membership inference the sample level, enabling more fine-grained analysis. Finally, additional research is needed fully explore the applicability of CoDeC across diverse model architectures and training regimes or proving theoretical guarantees for CoDeC scores.
5 RELATED WORKS
Contamination detection. Detection of training data contamination in LLMs is closely related to research on membership inference attacks (MIAs), which aim determine whether specific data sample was used in model training (Shokri et al. 2017). Early methods relied on training shadow models (Zhang et al. 202T] Yuan et al. 2023), which prohibitively costly for modern LLMs. simple approach detecting contamination is based thresholding loss (Yeom al., 2018)) perplexity (Carlini et al. (021). (Shi et al.
2024 Zhang ot al. 2024) improve over those methods by focusing on the most informative tokens. |Mattern al. (2023) Mitchell et al. (202) detect contamination by perturbing samples check for memorization. Unlike based approaches, CoDeC does not require model-specific threshold tuning and offer much better separability, which makes it particularly useful in practice.
9

---

Published as a conference paper at ICLR 2026
Many approaches rely on reference models for calibrating the entropy (Min al. 2024 Maini et al. 2024
Similarly, Carlini et al. (202T) proposes estimating entropy using zlib library. More recent approaches leverage held-out data to train a model-calibrated contamination classifier (Maini et al. 2024 Bar-Shalom al. 2025).
CoDeC in a way follows the general structure of reference-basee criteria, but uses the same model as reference, adapting to its internal confidence distribution.
Numerous approaches analyzed benchmark contamination in L.Mss. For example, Yang et al. (2023) show that simple n-gram overlap de-duplication insufficient, as paraphrases, translations, and variations of test samples still leak into training data. Golchin & Surdeanu (2024) observed that some models, when given only the name a benchmark, can recreate its questions. Oren et al. (2023) compare the loss when samples presented in canonical order versus random order, under the assumption that contaminated model will favor the former. Other approaches measure whether the model can reconstruct masked answers or even answer without being given the question (Deng et al., 2024, [Balepur al. 2024). However, such methods often suffer from low recall or rely on narrow assumptions about the targets, whereas CoDeC avoids these limitations entirely.
Recent studies prove that the problem of membership inference is inherently hard, especially in the era of LLMs trained on massive corpora and usually for just single epoch, rendering most prior methods ineffective (Maini et al. 2024: Duan et al. 22024] Carlini et al. 202T). Instead, Maini et al. (2024) proposed dataset-level approaches achieve meaningful results, but they require access held-out identically distributed samples. CoDeC follows the dataset-level setup, though it works without additional constraints.
Beyond attacks, several defenses have been proposed mitigate contamination detection. Tia et al. (2019) introduced adversarial filtering to reduce memorization. [Shejwalkar & Houmansadr (2020) explored selective forgetting using differential privacy and gradient masking, while Mireshghallah et al. (2021) investigated privacy-preseriing fine-tuning obfuscate membership signals.
In-context learning. LLMs are known to improve when prompted few-shot (Brown et al. 2020). The connection between in-context learning and training is widely studied relationship (von Oswald et al. 2023 Mosbach et al. 2023; Dai et al. (202). CoDeC builds upon this work, and extends the understanding of how in context examples impact generation confidence differently when the model has already seen the text
not.
6 CONCLUSSIONS
We introduced Contamination Detection via Context (CoDeC), simple yet very effective method for detecting and quantifying training data contamination large language models. By measuring how in-context examples from the same dataset affect model predictions, CoDeC distinguishes between datasets the model has memorized and those it has not. Our experiments show that CoDeC produces clear, interpretable contamination scores on wide range of models and datasets, exposing evidence of memorization and overfitting even in open-weigh models with undisclosed training data.
Unlike traditional membership inference methods, CoDeC requires no external references, dataset-specific tuning, or retraining, making it practical for large-scale, real-world evaluations. Its percentage- based scores are easy to interpret and integrate into benchmark reporting.
While high CoDeC scores (> 80%) indicate likely contamination, moderate values should be compared across models to separate dataset effects from memorization. Importantly, the scores do not indicate only strict training membership but capture also the influence from rephrased, augmented, or otherwise highly related data, making CoDeC practical tool for ensuring fair evaluation.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
In the supplementary material, we include notebook that enables running CoDeC on any chosen model and dataset. All models and almost all datasets used our experiments are publicly available on HuggingFace, making work fully reproducible. We strongly encourage readers to the attached code to evaluate contamination on data of their choice.
8
LLM USAGE
We used LLMs help us refine the wording of the manuscript, examine related works, and as coding assistants during experiment development. However, these uses were limited in scope and did not play a significant role the research itself.
REFERENCES
Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts abduction: How do Ilms answer multiple-choice questions without the question?, 2024. URL http::t/ts//xx...coo org. abbs/
2402. 12483
Guy Bar-Shalom, Fabrizio Frasca, Derek Lim, Yoav Gelberg, Yftah Ziser, Ran El-Yaniv, Gal Chechik, and Haggai Maron. Beyond next token probabilities: Learnable, fast detection of hallucinations and data contamination on Ilm output distributions, 2025. URL httttt:s://axiiiii oww.xiii.rrg..o.. abs 200..1110000
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2397 PMLR, 2023. URL httpp::/ppocceeinggg///oooooo//e mlr press http://2222//iieema232..3 htmll
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregres- sive Language Modeling with Mesh-Tensorfloww March 2021. URL
5281/
zenodo. 5297715
Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. Nemotron-h: A family of accurate and efficient hybrid namba-transforma models. arXiv preprint NeXiv:2504.03622 2025.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020. URI, http::/ts:///aar httppi.////a/i.o//o org
abs/ 14165
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song. Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. Michael D. Bailey and Rachel Greenstadt (cds.). 30th USENIX Security Symposium, USENIX Security 2021, August / 1-13, 2021, pp. 2633-2650. USENIX Association, 2021. URL http:::tttts.w..tt.. ottpt//////ww...eee.o.oo//////// wtt.//aalininiie htt.://aa...ooaaaanannetttrreare http://wwww..a.e.oo.o.....eeeeee

---

Published as a conference paper at ICLR 2026
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers, 2023. URL
https://arae /aaiivv/aaivvvv/aa org/ aww.////1/2 10559
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models, 2024. URL httttttt:////xxv..coo
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 1286-1305. Association for Computational Linguistics, 2021. doi: 10..05///8 EMNLL-MAIN....... URL https:///61..rg/10..
emn 1 i n. 98
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models?, 2024. URL / www.ii.....aaa.......ora http://bb//242////
C7841
Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. Does data contamination detection work (well) for Ilms? survey and evaluation on detection assumptions. NAACL, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding. Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.0000 2021.
URL httts::/arriv..or//abba.
00027
Shahriar Golchin and Mihai Surdeanu. Time travel 1lms: Tracing data contaminaiion in large language models. In The Twelfth International Conference Learning Representattinss ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. BRLhhttss: /tt.ptppppss//eoeaeieeeeeeeeeeee 2ttp://wwe..netttmmmmmoooooooeee
Dirk Groeneveld, Iz Beltagy, Evan Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, and David Atkinson al. Olmo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 15789-15809, Association for Computational Linguistics, 2024. doi: 10.18663/V1120444ALLLLLO
URL //doi.og//do.orgg
10. 10.18653/vll 20w11//222....2 ac woo.on.... 841
HuggingFaceH4 Team. Aime 2024 dataset, 2024. URL http:: https:://uugiiiga..c... co/datasets
MuggingFacetlo aimm__022 2024 Problems from the American Invitational Mathematics Examination 2024, accessible via Hugging Face coneentkeefrenne/llliiiieeeeelee
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversarial word substitutions, 2019. UtI.hhttss://arxiiii org. abs 909 00986
Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. LLM dataset inference: Did you train on my dataset? In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (cds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024,
12

---

Published as a conference paper at ICLR 2026
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf. Mrinmaya Sachan, and Taylor borg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison, 2023. URL TRL bttts::/aaxiv..oe abt////bb///a 2305. 18462
Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A. Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in nonparametric datastore, 2024. URL
abs////////////////////////// 2308. 04430
Fatemehsadat Mireshghallah, Huseyin A. Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg- Kirk-iirkpatrick, and Robert Sim. Privacy regularization: Joint privacy-utility optimization in language models, 2021. URL
httpps//azzad..oo///aa/////
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot mawhhin--gneraateee text detection using probability curvature, 2023. URL https://azxiv org abtaabs/s111 11305
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation, 2023. URL htttp:://axii..ottt/ abs/
7305.16933
Huu Nguyen, Victor May, Harsh Raj, Marianna Nezhurina, Yishan Wang. Yanqi Luo, Minh Chien Vu, Taishi Nakamura, Ken Tsui, Van Khue Nguyen, David Salinas, Aleksandra Krasnodebska, Christoph Schuhmann, Mats Leon Richter, Xuan-Son, Vu, and Jenia Jitsev. Mixturevitae: Open web-scale pretraining dataset with high quality instruction and reasoning data built from permissive-fir text sources, 2025. URL
https: a www.a.ivvvorroo..o org/ abs 2509.2531
NVIDIA, Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, et al. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025a. URL https: www.a. ortpg/////.../....//.//// 14444
NVIDIA, :, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabalesh- warkar, et al. Nemotron-h: h: A family of accurate and efficient hybrid tamba-transsormer models, 2025b.
Utthttss:://est
org/ abs 2504. 03624
opencompass Team. Aime 2025 dataset, 2025. URL
cotpaaeseso/ttaasss...eee opencompass / AIME2025 Problems from the American Invitational Mathematics Examination 2025 (AIME 2025-I & II), accessible via Hugging Face :oontee/Reereenneeaaciieeee:iiee
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. Proving test set contamination in black box language models, 2023. URL httLhttps://aaat wttp:/oo///.bbb//111/0111
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derezynski, Xingjian Du, Matteo Grella, and Kranthi Kiran GV et al. RWKV: reinventing rnns for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 14048- 14077. Association for Computational Linguistics, 2023. doi: 10. 18613//11/22233 FF661////000.FNII SUNDINGG-EMNNLPSO URL httphttts///do.orrg 1tw.1/.116611111222222
emnlp. 936
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. Association for Computational Linguistics, 2022.
13

---

Published as a conference paper at ICLR 2026
doi:
URL /FLLLtaa////oo.co///001133/////1
Sindings-emn.. 59
Virat Shejwalkar and Amir Houmansadr. Membersshi privacy for machine learning models through knowledge transfer, 2020. /ttL://////// htwpgg/ab/19990
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLpttpss:// /ttp:///tt////eeeeeooooo.o/////.
net htt.:/orrrmmmimmmm (og 3MQuNs
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models, 2017. URL /ttpp:s//aa2..../////////
Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in llms: how do we measure it and (when) does it matter?, 2024. URL https: http///xxxx..orra/////////// 03923
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for 2024.
language model pretraining research. arXiv preprint arXiv:2402.00/3:
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.0259% 2024.
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev,
Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent, 2023. URL
fatps://arxiin org/ abs 22122022211 07677
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, and Jianxin Yang et al. Qwen2.5 technical report. CORR, abs/2412.15115, 2024. doi: 10.48500AARXV.2412.5511 URL
http::///ooorg/1....00 http://.................2.2222..
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, and Feng Hu et al. Qwen3 technical report. CORR, abs/2505.009888 2025. doi: 1044850/ARXIV2255000022 https://aoi.org//1 http:/....4555/axiii1//// 2tt.alv..25500.03.0.
URL
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and lon Stoiea. Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL https::/arxiv..o org/ abs/
2311. 04850
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting, 2018.
01..0.9...666000 614
Wei Yuan, Chaoqun Yang, Quoe Viet Hung Nguyen, Lizhen Cui, Tieke He, and Hongzhi Yin. Interaction-level membership inference attack against federated recommender systems, 2023. URL https://arxiv
brq/abs/22n 2301. 10964
Jingyang Zhang, Jingwei Sun, Eric C. Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Helen Li. Min-k%+++ Improved baseline for detecting pre-training data from large language models. CoRR, abs/2404.02936 2024. doi: 1048550/ARXX.22444409966 URL https: http://ooo..gg///
48550/arXiv 429366
14

---

Published as a conference paper at ICLR 2026
Wanrong Zhang, Shruti Tople, and Olga Ohrimenko. Leakage of dataset properties in multi-party machine learning, 2021. URL https: // http:///wwwwaa.cc/////////0000/0
15

---

Published as a conference paper at ICLR 2026
A DETAILED EXPLANATION OF CONTAMINATION DETECTION VIA CONTEXT
A.1 KEY IDEA
Consider the following illustrative example. If mathematician trains for the International Mathematical Olympiad (IMO) competition by solving all available IMO problems, their knowledge becomes highly contaminated with IpO-specific problem characteristics. Beyond the genuine mathematical knowledge acquired, exposure to the IMO problem distribution introduces several specific priors, such as: all assumptions mentioned in the problem are necessary, the hypothesis requested to prove always holds, the problems are challenging yet formulated without advanced academic concepts, etc. These properties are unique to IMO problems but not to general mathematical questions and texts. Therefore, leveraging such priors results an unfair measure of onc's general mathematical skills.
Now, suppose this mathematician is given free access IMO problems during the competition. If they have already learned to solve those problems, such context is no longer useful for further learning. However, if the mathematician has learned from other sources and has never seen any IMO problem, they would benefit greatly from understanding the specifics of these problems, their structure, and implicit assumptions.
Following this example, Contamination Detection via Context (CoDeC) measures the contamination score by evaluating how access to the target dataset affects the model's predictions. For each datapoint in the dataset, CoDeC samples a few in-context examples from the same dataset and compare the average logits with and without this context.
the model has not used the target dataset for training, it should benefit from additional context, as these examples contain valuable information about the data distribution. Even if not directly related, they may share similarities in structure, style, vocabulary, topic, or other implicit common priors. Conversely, if the model was trained on that dataset, it already knows these priors, making the provided context less beneficial. Furthermore, if the model is overly confident due to memorized patterns, such as specific word sequences or frequent occurrences, introducing additional context (likely also memorized) should disrupt these patterns and negatively affect the confidence.
See the analysis in Appendix C.2 for empirical evidence of those observations.
A.2 CODEC PIPELINE
Following the key idea described above, the complete pipeline of CoDeC consist of 4 simple steps:
1. Baseline prediction: For each datapoint æ in the suspect dataset 'D obtain the model's average log-likelihood on the consecutive tokens of z.
2. In-context prediction: Sample n additional examples I1. .... In from D \ {x}, prepend them to x (creating sequence x1||- _r||-.//m ()), and obtain the model's predictions on x this new context. Focus solely on the probability of tokens in xr, ignoring the context examples. While any number of context examples can be used, in our experiments we used one.
3. Score computation: Compute the difference in confidence A(x) logprobin (o)tex())) logprob (z). Since the context is sampled randomly, the value of A A(x) is subject to some variance. To achieve higher statistical significance, though even a single seed provides meaningful scores. Additionally, when computing the average log-likelihood, We ignore first 10 tokens to avoid the influence of switching between texts.
average the A(x) values over 5 seeds,
4. Aggregation: Repeat the above for all x E D. The contamination score for the dataset is then
N
ScoDer (D)
[A(x,) < 0
N
16

---

Published as a conference paper at ICLR 2026
where 1l is the indicator function.
summary, to compute CoDeC scores, we compare the model's confidence with and without additional context for each sample. The final score represents the percentage of samples for which the influence of additional context is negative.
A.3 PROPERTIES CODEC
A.3.1 THEORETICAL PROPERTIES
CoDeC by its design has several desirable properties that make it useful in practice:
Score as a percentage. CoDeC returns contamination score percentage of dataset elements indicating contamination, making it intuitively understandable. Unlike classical approaches that compute scores in open ranges requiring references and scale knowledge, percentage points are easily grasped without in-depth analysis. Even the intuitive notion of probability of contamination, while not fully grounded, can be useful here.
Works with any dataset. CoDeC can be computed for any set of strings, making it broadly applicable. In our evaluations, we used various data sources: standard training datasets (e.g.. Wikipedia, Common Crawl), code (e.g., GitHub), QA benchmarks (e.g., MMLU, GPQA), math benchmarks (e.g., AIME, MATH-500), PDFs (e.g., ArXiv), books (e.g., Project Gutenberg), websites (e.g.. global news, Amazon Reviews), files (e.g., Linux syslog), and more. While CoDeC is designed primarily for datasets, is straightforward to transform a single text source into a dataset by splitting it into parts.
Works with any gray-box model. Any model that outputs logits for a given text (including most models available HuggingFace) can be used to compute CoDeC. The method is independent of the model's training specifics and architecture. It can even be applied to fully black-box models by empirically estimating target token probabilities, though this increases score variance unless assume multiple repetitive calls and model stochasticity.
CoDeC does not require model-specifie or dataset-specific parameters. Unlike classical MIA methods, which need a tuned classification threshold and deep model knowledge or external ground-truth data, CoDeC can applied off-the-shell to any model and dataset.
Parameter-free.
Computationally efficient. To compute CoDeC scores, the model runs twice with at most twice longer samples, introducing minimal computational overhead. Using as few as 1000 samples already yields precise scores (see Figure hence it be applied even large corpora. Additionally, in-context learning much faster than methods requiring finetuning or shadow model training.
A.3.2 EMPIRICAL PROPERTIES
Furthermore, during experiments observed the following empirical properties of CoDeC:
Robustness to text cropping. CoDeC scores remain stable when using partial training prompts. This property proves particularly valuable when evaluating single, long text sources (books, articles) that should be split into dataset components.
Sensitivity to formatting variations. Artificial formatting changes (added labels, modified whitespace) reduce contamination scores even for training data. Understanding that sensitivity is crucial for reliable
17

---

Published as a conference paper at ICLR 2026
QA benchmark evaluation, where labels like Question: Answer: instruction prompts are common. Incorrect labels (those not used during training) decrease scores, as expected: in-context samples provide formatting information that artificially increases prediction confidence. Furthermore, incorrect labels may break memorization patterns themselves. To ensure meaningful evaluation, the suspect dataset should focus exclusively on text portions guaranteed to appear during training (e.g., question text only), independent of training-speciffi formatting. As shown in Section 3.3 CoDeC captures the contamination even in cropped samples, so uncertain parts may be safely excluded.
Impact of data diversity. Training datasets consistently yield scores near 100% regardless of their properties. However, unseen dataset scores vary from 0% 60%, depending dataset characteristics. CoDeC assumes that additional context from the same distribution provides some additional information, e.g. text type, style, vocabulary, domain knowledge, etc. In large, diverse datasets with unrelated samples, minimal information sharing between datapoints occurs. Hence, additional context may not improve predictions and can act as random noise, potentially yielding CoDeC scores up 60%, even for unseen data. Our experiments suggest that this effect may be mitigated using more context samples.
Larger models use less memorization. Our main evaluation (Figure 3 included models from 410M to 56B parameters, where training data could be verified. While results remain consistent across architectures and sizes, larger models within families show slightly lower CoDeC scores. This trend intensifies at larger scales. For instance, aaama-Nemotron-Nare 4B exhibits high contamination across multiple benchmarks, while Llama-Nent Newotron- Cottotton--Utraaaa 253B maintains all contamination scores below 20%. Although these models share family, their training processes and data compositions differ, however, these differences alone cannot explain the contamination disparities.
This size-dependent behavior aligns with expectations. CoDeC measures reliance on memorization instead of general knowledge. Larger models possess greater capacity for genuine knowledge acquisition, reducing overfitting to individual datapoints. Consider an illustrative example: when solving algebraic equations, children often rely on pattern memorization, while expert mathematicians apply fundamental skills even for previously encountered problems.
Adversarial cases. While CoDeC provides robust contamination scores for most datasets and models, adversarial constructions remain possible. For instance, a dataset containing 1,000 identical (or near-identical) samples produces low CoDeC scores regardless of training exposure. Constructing datasets with arti artificially high contamination scores is more challenging but possible through combining several unrelated, diverse data sources. See a detailed analysis in Appendix C.7
These edge cases do not compromise evaluations on standard benchmarks, as genuine datasets remain unaffected by such artificial constructions. Since evaluation data selection remains under user control, models cannot exploit these constructions to bypass detection.
CoDeC applies to any causal model by design. We evaluated over 40 models across various sizes and architectures. One notable exception emerged: GPT-OSS 20B consistently produces contamination scores exceeding 99% across all datasets. Investigation revealed heavy optimization for chat and reasoning tasks that impairs standard language sequence modeling. Even when provided identical text as context, confidence decreases due persistent attempts to terminate sequences and transition to thinking chat-like dialogue patterns. Addressing that issue remains for future work.
A.4
WHY DOES CODEC WORK?
The central idea behind CoDeC is to measure whether a model has internalized a dataset's specific priors, their presence is a clear indicator of training data contamination. These priors are not limited to exact
18

---

Published as a conference paper at ICLR 2026
string memorization but cover a wide range of cues, such as stylistic patterns, characteristic vocabulary, or common structural templates. While a model may lack the capacity to memorize every single training example, the manifold of these general cues significantly narrower and can be memorized more easily. CoDeC is designed to detect if the model leverages this learned manifold.
Our approach uses in-context learning as an efficient proxy for finetuning- Consider the standard finetuning process: a model performance improves significantly during the first epoch on a new dataset, with smaller gains in subsequent epochs. Similarly, if we could finetune a model on target dataset for contamination detection, contaminated model would improve much more slowly than non-contaminated one because it has already learned the data distribution. However, performing actual finetuning for every evaluation is computationally prohibitive, especially for large models. CoDeC achieve a similar outcome by using in-context learning, which is a negligible cost. Hence, in essence, CoDeC measures the remaining learning capacity for the target dataset. If the model has already been trained on the data, it has little capacity left learn, and its performance will not significantly improve when presented with in-context examples from that dataset.
This behavior can be also understood through the lens of the loss landscape. Contamination is often linked overfitting (Maini et al.) 2024), where the model settles into a sharp, narrow local minimum in the loss landscape for the training data. For unseen data, the loss landscape tends to be flatter. Our experiments suggest that in-context learning acts similarly to finetuning with a high learning rate. This makes the process highly sensitive to the local geometry of the loss manifold. For contaminated sample, the model is already a steep local minimum. Hence, taking even single step likely to exit this minimum, resulting in worse performance. Conversely, for an uncontaminated sample where the loss manifold is flat, the same step will likely move the model towards a region of higher confidence, improving performance at least slightly.
Finally, CoDeC relates to reference-based MIAs, which often use external models datasets to calibrate the difficulty of a given sample. CoDeC shares a similar structure but introduces a crucial distinction: it relies self-reference approach. Instead of using external data for calibration, CoDeC uses the model own predictions as the baseline. This design choice makes our scores independent of the availability of external resources and ensures that the calibration is based on specifically to the properties and knowledge of the model being investigated.
A.5
How TO INTERPRET CODEC SCORES?
The CoDeC score a measure of how much model's predictions rely on memorized patterns rather than genuine reasoning. Importantly, it does not indicate strict membership of dataset in the training corpus. Instead, reflects whether the outputs are predicted based on memorized internal distribution priors ssmething that can happen if the model was trained on identically distributed or closely related data (c.g. artificially generated). From another perspective, a lower contamination score indicates greater remaining model capacity to learn from the target dataset.
In practice, our experiments reveal two complementary ways to interpret CoDeC scores: absolute evaluation and reference-base based comparison.
A.5.1 ABSOLUTE SCoRE INTERPRETATION
Because CoDeC scores naturally fall between 0% and 100%, they can be compared across datasets and models without model-specific scaling or parameter tuning. Empirically, we found a consistent pattern:
Scores above 80% were measured for nearly all training datasets we used for experiments, indicating strong contamination evidence.
19

---

Published as a conference paper at ICLR 2026
Scores below 60% were measured for nearly all unseen datasets. In general, they show no evidence of contamination the model is likely reasoning based on general knowledge rather than memorization.
Scores in the 60%-80% range are ambiguous: they may be due to partial contamination or training on related distributions.
Thus, while an absolute threshold (>80%) a strong indicator, values in the intermediate range require more nuanced analysis.
A.5.2 REFERENCE SCORE INTERPRETATION
Absolute scores alone only partially capture variations in dataset properties like diversity. For example, even a non-contaminated model might score relatively high broad, highly diverse dataset. To adjust for such effects, the scores should be compared across multiple models on the same dataset.
The most reliable approach is to include least model that is known be non-contaminated with the target dataset (e.g. older model like Pythia). If all models show similar contamination scores, this suggests no substantial memorization specific any model, but rather dataset-specific level of the score. However, if a score of the model stands out as an outlier compared to reference models, this strongly suggests higher reliance on memorization.
Choosing reference models of similar size and architecture further helps ensure that differences in scores point to contamination rather than general learning capacity.
A.5.3 INFORMED MODEL COMPARISON
Contamination is not inherently negative. For example, models designed to solve advanced math problems are expected to be trained extensively on math datasets for excellence, naturally resulting in higher contamination. This is similar to top IMO competitors who have solved thousands of problems, many from previous competitions, to build their expertise. However, when comparing two models with similar performance, the with lower contamination likely more adaptable and possesses deeper, more general capabilities. Just as a person who excels at IMO without targeted training demonstrates broader intelligence, a model with less reliance on problem-specific priors is usually more capable.
CoDeC measures the remaining learning capacity for the target data distribution, indicating how much room a model has to improve with possible further task-specific tuning. This way, CoDeC allows to understand not only the raw performance of each model, but also the sources of that performance - how much it relies on genuine knowledge and how much on data-specific priors that not necessarily generalize to other related setups.
A.5.4 PARTIAL CONTAMINATION
Direct inclusion of benchmark in training is not the only source of contamination. Highly related data- such paraphrases, augmented samples. identically structured content, additional examples can also compromise evaluation, making observed performance less trustworthy. In practice, questions from widely used benchmarks often leak into public sources such as forums, blog posts, and educational material.
As shown in Section 3.3 (Figure 7). CoDeC detects contamination arising from training on related content. Scores should therefore be interpreted in terms of how the training data relates to the target dataset relationship that cannot be captured by classical n-gram overlap checks, commonly used in practice (Nguyen et al. 2025)
This explains why some datasets never explicitly seen during training still reach CoDeC scores as high as 60% (see Figure 13). For example, in experiments the highest-scored unseen datasets were Amazon
20

---

Published as conference paper at ICLR 2026
product reviews, the Colonial Memories book from Project Gutenberg, and Global News articles. Models trained on the Pile dataset have direct exposure to Project Gutenberg and Books3, making Colonial Memories strongly distribution-relatee even without being part of the training set. Likewise, news and product reviews are common in large web corpora, high contamination scores for them are not surprising.
Importantly, CoDeC still distinguishes these cases: data directly included training consistently score near 100%, while related but unseen data get high but noticeably lower scores.
A.5.5 BEST PRACTICES
For robust contamination assessment, we suggest combining absolute thresholds with reference-based comparisons. High absolute scores (>80%) should be treated as contamination red flags, but significant deviations from other models' CoDeC scores can be equally alarming. By viewing CoDeC scores not as binary labels but as indicators of memorization intensity, the results serve useful indicator both for ensuring a fair model comparison on benchmarks, and reliable model quality assessment during training.
We also suggest that, among models with equal benchmark scores, those with lower CoDeC scores should generalize better.
21

---

Published as conference paper at ICLR 2026
B EVALUATION SETUP
To make the main evaluation meaningful, we have models for which know the training data, and prepare additional datasets that were certainly not used for training. Following standard practice, we ensured this exclusion by using data published only after the release of the training datasets, climinating the risk of data leakage. In each case, we selected broad range of data types, sources, levels of diversity, and other characteristics ensure comprehensive coverage of possible data properties in evaluation.
For each dataset, we selected 000 random samples for evaluation. If the data source was continuous stream of text (e.g. book or an article), we split it into 600-characters chunks and treated dataset.
Since the selection of both training and unseen datasets is inevitably somewhat arbitrary, we emphasize that the results in this paper reflect all evaluations we have conducted, including internal experiments. Given that the pipeline is simple and lightweight, we strongly encourage readers to run it themselves and confirm the reliability of CoDeC on datasets of their choice.
Because most LLMs available today do not disclose their training data, our model choice is necessarily limited. Nevertheless, we selected the following families with full access to their training datasets:
Pythia Biderman et al.
2023).
trained on the Pile dataset (Gao et al. 202
GPT-Neo Black et al.
2021
trained on the Pile dataset.
RWKV-4
Peng al.
2023)
trained on the Pile dataset.
OLMo Groeneveld et al.
2024).
trained on the Dolma dataset (Soldaini al. 2024).
Nemotron-v2 (NVIDIA et 2025a), trained on the Nemotron-Ce dataset (Su 2024
Nemotron-H Blakeman el al.
2025 trained on the Nemotron-CC dataset.
B.1 TRAINING DATASETS
We used the following sets training data examples:
The Pile
HackerNews
Wikipedia
GitHub
ArXiv
DM Mathematics
Pile CommonCrawl
PubMed Central
Full Pile
Wikipedia Music (a low-diversity subset of Wikipedia containing only music-related articles)
GitHub Licenses (a low-diversity subset of GitHub containing only license comments)
Since the Pile dataset not available for direct download, we used the samples provided the
iamgroot 42.00t222/// 42/mimi dataset.
22

---

Published as conference paper at ICLR 2026
Dolma
C4
Wikipedia
Pes2o v2
Reddit v5
Stack v4
CommonCrawl head
CommonCrawl middle
CommonCrawl tail
Nemotron-CC
Wikipedia
StackExchange
ArXiv
CommonCrawl 2024 (high quality)
CommonCrawl 2020 (high quality)
CommonCrawl 2020 (low quality)
CommonCrawl 2019 (medium quality)
CommonCrawl 2016 (medium quality)
CommonCrawl 2014 (low quality)
CommonCrawl 2013 (high quality)
B.2
UNSEEN DATASETS
For unseen datasets, we used the following sources:
Popular benchmarks. We evaluated the models on the following benchmarks. For each model family, we used those that were published after the training dataset release date ensure they were unseen:
For models trained on the Pile: gsm8k, GPQA Diamond, IFEval, HumanEval, FRAMES, AIME 2024, AIME 2025, LiveCodeBench v1, LiveCodeBench v5, BFCL v3, BBQ, RewardBench v1,
RewardBench v2, and MATH 500.
For OLMo models: FRAMES, AIME 2024, AIME 2025, LiveCodeBench v5, BFCL v3, Reward- Bench v1, and RewardBench v2.
For Nemotron-H, used only AIME 2025.
Project Gutenberg. We used three books added to Project Gutenberg after April 2025:
Colonial Memories
Jibby Jones : story of Mississippi River adventure for boys
The Corbin necklace
23

---

Published as conference paper at ICLR 2026
Datasets from HuggingFace.
Nickyyhict//glooill-eee--dded
A recent website.
An article with Ukrainian conflict updates:
https:/www.uuderstaadnnnaaa...
Self-created data.
A document describing this project.
A transcript from meeting.
Linux syslog file from computer.
Internal slack channel log.
B.3 DATASETS FOR ABLATIONS
For the ablations (Figures 9a 9b and experiments with finetuning (Figure 6), used the following representative 10 datasets:
Training data (from the Pile)
HackerNews
Wikipedia
GitHub
ArXiv
Full Pile
Unseen data
Colonial Memories (Project Gutenberg)
Ukrainian conflict updates
Global News
gsm8k
Amazon reviews
B.4 AUC COMPARISON
The Area Under the Receiver Operating Characteristic Curve (AUC) is a standard metric for evaluating binary classifiers. It measures the probability that randomly chosen positive example is ranked higher than randomly chosen negative example, making it threshold-inddependent and robust to class imbalance. S+ the set of scores for positive examples and S for negative examples, then:
24

---

Published as conference paper at ICLR 2026
AUC =
Σ
Σ
>8) 1(s+ ES
1(s
0.5 1(st = =
S+ S
In the context of Membership Inference Attacks (MIAs), AUC is widely used to quantify the classifier's ability to distinguish between training samples (positives) and unseen samples (negatives).
A key difference between setting and typical MIA evaluations is granularity. In most prior work, contamination scores are computed per sample, and the AUC reflects the quality of sample-level classification. In contrast, our method focuses on the dataset level: compute single contamination score per dataset and evaluate the model's 's ability to classify entire datasets as seen or unseen. This approach better reflects the intended use CoDeC, which is designed provide interpretable, aggregate measures of contamination for benchmarks and corpora rather than individual samples.
As shown in Figure CoDeC achieves near-perfect separation between seen and unseen datasets, leading to dataset-level AUC scores close to 100%. This demonstrates that CoDeC is highly effective at capturing training data contamination way that aligns with MIA evaluation traditions.
25

---

Published as conference paper at ICLR 2026
C EXTENDED EXPERIMENTS
C.I BASELINES
In our analysis, compare with the following general-purposs baselines:
Vanilla loss. Models are trained to optimize the loss training examples. Hence, straightforward
way to identify training data is to score the samples based on the average token loss.
Min-K Introduced by (Shi et al. 2024), Min-K% builds the observation that training samples do not have tokens with very low probability. Hence, the criterion computes the score based on the loss measured only the least-probable tokens in the sequence.
Zlib ratio. Loss-based methods, while effective in some settings, cannot distinguish contaminated data from genuinely easy data, as both result in low model loss. To fix this problem, the Zlib ratio method (Carlini et al. 2021) normalizes the score using sample entropy, estimated as the Zlib compression rate.
C.2 INFLUENCE THE CONTEXT ON MODEL LOGITS
C.2.1 CHANGES IN LOGITS WITH CONTEXT
(a) Changes in model confidence an unseen dataset (gsm8k).
(b) Changes in model confidence a training dataset (ArXiv).
(c) All samples from GSM8K (unseen), averaged.
(d) All samples from ArXiv (training). averaged.
Change in model log- probabilitiess with and without context for the Pythia 1.4B model, evaluated on samples from unseen dataset (gsm8k, TOa) and training dataset (ArXiv. 10b). The two bottom plots show the differences averaged over the whole dataset.
Figure 10:
We observed that models react differently to context depending on whether the dataset seen during training or not. Specifically, we pass the context by sampling an additional text from the dataset and prepending it to the target sequence (connected with two newlines). Then, compare the log probabilities for each token in the target sequence with and without the added context. For better stability, we ignore the first 10 tokens of the target sequence to exclude the influence of transitions between texts.
26

---

Published as a conference paper at ICLR 2026
If the dataset from which the samples were taken (both the target text and the context) was seen during training, the log probabilities consistently decrease. However, for unseen datasets, we observe consistently higher estimated probabilities of the correct tokens in the target text. Hence, CoDeC determines whether the samples were seen during training based on whether the cumulative difference is positive or negative. Note that since we check only the sign the difference, the scale those fluctuations is not important. Therefore, CoDeC works for each model regardless of its internal properties.
Figure Toa shows the logprob differences for the text "Ai the beginning of an academic year, there were 15 boys in a class and the number of girls was 20% greater. Later in the year, transfer students were admitted such that the number of girls doubled but the number of boys remained the same. How many students are in the class now?" with and without the context text "Ten adults went Adult tickets are S8 each and the total bill was $124. How many dollars is one child's 's ticket?". Both texts are sampled from the gsm8k dataset, which was not used for training the Pythia models. Despite the texts having different topics, they share a similar structure; hence, we observe that most tokens in the target text benefit from the added context.
ball game with eleven children.
In contrast, Figure shows the logprob differences for the text "We show that a phase-only spatial light modulator can be used generate non-trivial light distributions suitable for trapping ultracold atoms, when the hologram calculation is included within a simple and robust feedback loop that corrects for imperfect device response and optical aberrations." with and without context text "We consider one source of decoherence for single trapped ion due to intensity and phase fluctuations in the exciting laser pulses. For simplicity we assume that the stochastic processes involved are white noise processes, which enables us to give a simple master equation description of this source of decoherence." ". Both texts are sampled from the ArXiv dataset within the Pile corpus, which was used to train Pythia. Although both texts share a similar structure (both opening abstracts), both are related to physics, and both similar vocabulary, nearly all log probabilities decrease when the context is provided due to disrupted memorization.
The described behavior is highly consistent, demonstrated in main experiments (Figure
moxsing
(a) Changes in model confidence on an unseen dataset (IFFval).
Changes in model confidence on a training dataset (Wikipedia).
-25
(c) All samples from IFEval (unseen), averaged.
(d) All samples from Wikipedia (training), averaged.
Figure 11: Additional plots of changes in model log probabilities with and without context.
27

---

Published as conference paper at ICLR 2026
C.2.2
ATTENTION PATTERNS
Siew Selowayy
biberay
Attention patterns before training.
Attention patterns after training.
Figure 12:
Change attention patterns after training. Each row represents token saliency maps for the next token in the sequence, given the prefix. Yellow/green points indicate high reliance, while dark blue points indicate low reliance. As the training progresses, the reliance polarizes most predictive tokens become strengthened, less predictive tokens become completely ignored.
Figure 12 show how the attention patterns of the model change after exposure to the data. The chart shows attention patterns for the consecutive tokens (each row represents how much the corresponding token attends to previous ones; the lighter the color, the stronger its reliance). The model (Pythia 410m) evaluated on 3 random problems from gsm8k (with initially very low contamination) and finetuned for 2 epochs on gsm8k (further training doesn't change much). The contamination rises to 100%. Initially (before training), the model attends to some tokens more and some less (as usual), but generally many tokens attended to (multiple points in each row are greenish). The attention covers mostly the current question, but somewhat extends also to previous context examples. As the training progresses, the attention patterns sharpen. Tokens that were attended strongly get even stronger, and less important tokens get even weaker. The reliance on the context nearly completely removed, though some minor influences from the significant tokens remain visible. It happens mostly during the first steps of finetuning. Later steps strengthen the effect a bit, but generally bring little further changes. This behavior consistent across different runs and samples.
con-contaminated models rely multiple cues, both in the current sequence and in the context. Hence, they can utilize the additional information given. Contaminated models sharply rely on selected tokens usually these are not spurious words that serve nothing but triggering memorization, but these are genuinely informative tokens that initial model also strongly paid attention to. Just the reliance becomes overly strong and exclusive. Contaminated models tend to ignore the context (note that this is an emergent behavior). Memorization patterns do not sharply transfer from context to other samples; however, their influence not completely ignored, contributing at least bit lower confidence.
28

---

Published as conference paper at ICLR 2026
C.3 PER-DATASET SCORE DISTRIBUTION
Score Distibutionn per Detaset Acrses Models
Datasel D venffy
cicet desoraiion Unier cootec ipaites
fexs
ARE 2014
2055
Ikrundua
Man 5C0 (eztamm)
Nowar Bondh
Gena fore Jel
Catanber:
Ronard bench
Aesuse Resios
Gitku (Rceas))
Witooda (raac)
Diarr
Figure 13: Distribution of CoDeC scores dataset, computed for models trained on the Pile. The last 10 datasets parts of the training data. Among the unseen datasets, the largest scores receive those that more related the subsets of the Pile.
C.4 CONTAMINATION DURING TRAINING
CoDeC Scores During Training
Dataset type Treiiing dalasels
Detasets
Doima Redst
Amazon Reviews
Contamination Score
80%
Unseen datasets
polma stack ya
Dolma CC En Middle
Global News (doscriptionil
Noward surch
Dolma CC En Head
BFCL v3
Dolma DC in
Reward teech
60%
Dolma D4
Sutenberg Nucklace
Dolma Simple
Rames
Dolma Feeda 92
Global Newx
LvcCodeBencce
Jebyy Janee
20%
Ukraine Contrct Updates
AIM 2014
Slack
codlc: descriction
10a
303
108
Training steps
Figure 14: Changes in CoDeC scores over the course of training OLMo 7B. Already after 10k training steps (about 2% of the full training), the contamination step 0 all scores oscillate around 50%, as a random model has no ICL skills all that however learned even before step 500, hence all scores immediately go to 0 as the context becomes helpful, but nothing was memorized yet.
have nearly converged to their final values. At
29

---

Published as conference paper at ICLR 2026
C.5 CONTAMINATION TRANSFER
Contamination transfer from other datasets
80%
Contamination
60%
Koniie cinpe (195))
Randor capa
trilting Jruan 113
nega
Ni mnmiiinee and urienn (2944 Ast maltonnng
Nol coniiee bnteeS (191
centiee (791
20%
randamiaga
randram Inder (20%)
50
+ Finetuning batches
69
19
+P
Finetuning
Finetuning batches
Figure 15: Contamination transfer to MMLU from related distributions, computed for the Pythia 1.4B model.
Contamination transfer other datasets
100%
Add leders (2%)
Contamination
60%
Add rancor hites 2200 Add ledira
Kentuiing danille
40%
Randim 17550
Kardon clopi
milling and anteen (294)
CFBA
traiting and 125%)
Aueliurr CA
Asd rendomiags (3)
rendomiags
8
Finetuning
Finatuning batches
Anetuning batches
Figure 16: Contamination transfer from MMLU to related distributions, computed for the Pythia 1.4B model.
Contamination transfer rephrases
Contamination rephrases
100%
80%
CoDeC Score
CoDeC Score
60%
40%
40%
20%
GSHIK Art Nebioio
20%
Restrass GGN3K
Restendos Whisesta
09
Restradie Arda
10
120
100
120
Finetuning batches
Finetuiing batches
Figure 17: Contamination transfer between benchmarks and their rephrased versions, computed for the Pythia 1.4B model. Rephrases were obtained using GPT with prompt "Please rephrase the following text, but keep the meaning the same. Do not change the meaning of the text. Answer with just the rephrased text, other text comments." ". In both cases, the contamination is clearly detected.
30

---

Published as conference paper at ICLR 2026
Finetuning on Pile with addition
Finetuning on OOD datasets simultaneously
100%
CoDeC Score
CoDeC Score
HELJI
Goter boy
207
076
Amrzzon revicss
1000
100
500
finetuning
Finetuning
Figure 18: CoDeC scores for a model trained on broad data mix. In the first experiment (left chart), we mixed multiple subsets of the Pile and the corresponding benchmark, simulating standard training corpus. In the second experiment (right chart), mixed multiple unseen datasets and evaluated the progress of CoDeC scores on each component of that mix. In each case, the contamination is clearly detected.
C.6 RELEASE-ALIGNED VALIDATION ON ANNUAL BENCHMARKS
We further validate CoDeC in setting where training corpora are available. Many benchmarks released anmually; CoDeC captures reliance on memorized priors, then models whose training cutoff predates a given benchmark year should exhibit higher CoDeC scores those pre-release years than post-release years.
For each model we define a pre/post partition of benchmark years based on public release timelines (see Table 2 For each (model, year) pair compute the dataset-level CoDeC score and summarize per model across years. Across six models spanning three families, we observe a consistent Pre >> Post pattern (Figure T9p: median CoDeC drops by 5-15 percentage points after the benchmark's s release, with per-model Wilcoxon tests significant after FDR correction.
100
Target release
80
Post-releace
CoDeC (1)
60
gemma 3 4b it
gemma 12b It
Mota Liams 3.1 instruct
Qwen? 88
gemma
Model
Figure 19: Release-aligned CoDeC scores (%) by model. For each model, benchmark years are grouped into Pre-release (blue) vs. Post-release (orange) according to the model's training cutoff (see Table Boxes show the distribution Scare across years; medians decrease from pre post for all models, consistent with CoDeC detecting reduced reliance memorized priors the benchmark is nut-of-distribution relative the training window.
31

---

Published as conference paper at ICLR 2026
Table 2:
Pre- and post-release IMO years per model (see (Hugging Team 2024
opencompass Team
2025 ) used for the splits in Figure
Model
Pre-release years
Post-release years
Meta Llama 3.1 8B Instruct
2022
2023, 2024, 2025
Llama 3.2 3B instruct
2022
2023, 2024, 2025
Gemma 1B
2023
2024, 2025
Gemma 4B it
2023
2024, 2025
Gemma 12B
2023
2024, 2025
Qwen3 8B
S 2023
2024, 2025
C.7 ADVERSARIAL DATASETS
CoDeC scores provide reliable and robust contamination estimate, but just like any other metric they prone to adversarial or degenerate cases. is possible to construct artificial datasets for which CoDeC scores will be 0% regardless of the model and its true contamination level. A carefully designed dataset allows to achieve higher contamination scores, though in that case it's much harder to achieve. It means that constructing false positives is not easy even on purpose, which desired property.
Observe that it is clearly possible to construct benchmarks that yield universally low or universally high scores regardless of the model evaluated. Likewise, is possible construct adversarial datasets for CoDeC. However, such evaluations serve no practical purpose and their existence does not compromise the regular evaluations.
Adversariial/degenrrate datasets
DleutherA//ytthia-laae 1.4b
RWKV/nwkr-44430n-pppm
allenai/OLMo- IB-hf Mo-
Qwen/Qwen3--n 1.7B
Dataset
One repeated (ID)
0
One text repeated (OOD)
0
One text repeated with noise (ID)
0
One text repeated with noise (OOD)
0
Random sequence of words
0
0
0
0
Adversarial mixture
41
55
37
57
Table 3: CoDeC scores for adversarial and degenerate datasets, carefully crafted to bias evaluations.
C.7.1 ACHIEVING UNIVERSALLY LOW SCORES
CoDeC scores are sum of two opposing effects: confidence increase due to the additional information provided in the context, and confidence decrease due to disrupted memorization. one of those effects artificially amplified, the scores may become universally lower or higher.
The easiest way to amplify the confidence increase is to inject overwhelming redundancy the dataset. For instance, if the dataset consists of just single text copied 100 times, the objective becomes trivial, the target text is always given precisely in the context. Each model will hence increase its confidence, regardless of whether that text actually used for training not. And as shown in Table 3 is indeed the case.
32

---

Published as conference paper at ICLR 2026
How severe is that property in practice? As shown in Figure 20 even when reduce the dataset as few 8 distinct samples repeated 16 times each, the CoDeC score remain barely affected. Only reducing the diversity even further visibly reduces the value. practice, benchmarks exhibiting this level of repetitiveness are problem themselves.
CoDeC Score Number of distinct samples
CoDeC Score
0.6
0.4
128
Number of distinct samples (out
Figure 20: CoDeC scores on repetitive datasets. The x axis is the number of distinct samples in the dataset, indicating repetitiveness factor of 128/x. The scores are evaluated for the Pythia 1.4b model on Wikipedia.
We also experimented with an extremely degenerate case, in which the dataset consists random word sequences. The scores equal 0 even having no related meaning structure turns out to be dataset-wide helpful property.
C.7.2 ACHIEVING UNIVERSALLY HIGH SCORES
While it is very easy to amplify the positive influence, amplifying disruption is much harder. One could think that simply augmenting the previous approach by injecting small random noise (e.g. single additional random word) would be enough a model convinced that the task is to copy the context should receive extremely high penalty when failing on the injected word. In practice it not the case noisy words indeed penalize the model, but the advantage gained on other tokens always overtakes.
A more effective approach leverages the fact that unrelated context does not successfully improve model's s confidence on the target text (see Figure 21). Hence, if we mix several unrelated data sources, the sampled contexts may turn out be less relevant and fail increase confidence, resulting in high CoDeC estimates. Indeed, we mixed: IFEval, Linux syslog, GSM8K, and meeting transcript. While each evaluated model gets 20% on each of those datasets separately, they get scores of around 50% on the mix. By a careful choice of the subsets, we believe the scores may rise even further.
Such mixes may appear practice. For instance, MMLU consists of multiple distinct topies, much of which are completely unrelated. However, even in such cases the scores usually do not exceed 50% and remain consistent across different models.
C.7.3 DBALING WITH ADVERSARIAL CASES
Existence of adversarial datasets is not an issue itself, since the choice the dataset for evaluating contamination is always the user side. To counter the influence of unrelated contexts, informed sampling methods
33

---

Published as conference paper at ICLR 2026
Contect Context-dependder dependent Centmmmittinnn SCO
giferbrruattona www.goooooouaaeeeooeooo Liamonee
welgnctt Jent no
pasvee centre norwn, ou posaeped
Nacoonics, Poonuut PeAAu
Garce rann 13 on TRAVULEd
desterniin and srgee yeeree coneiiiii cnteen sen
2000 1000
air proien
Figure 21: CoDeC ablation: impact of the context distribution. Each cell represents what would be the "ablated contamination score" if for evaluating dataset Y context would be sampled from dataset X. and not from it done CoDeC. Providing unrelated context brings relevant information, hence the confidence usually drops. Note that those scores are not possible to reproduce within the standard CoDeC pipeline.
(c.g. KNN) may be useful, as they allow to keep the computations within data clusters, making the scores independent of data mixes. Additionally, we note that even though certain data properties may amplify the positive negative context impact, the main driving forces of CoDeC remain valid, so truly contaminated models still should exhibit considerably higher scores than other models. Hence, is important to include baseline models in comparison, as suggested in Section 3.5.
C.8 KNN- BASED CONTEXT SELECTION
Selectin
Draant
FACC-
Uneen (1/11))
(M)
MATHIIO
Dalavit (MLL))
Cankral
Condition
(b) Cross-dataset context: CoDeC drops when con- from similar dataset (MMLU) and remains higher with less similar context (Pile, MATH-500).
(a) Within-datasel context: similar (KNN-1) context lowers CoDeC scores, while random reflects the dataset-average baseline.
Figure 22: CoDeC scores under KNN-based context selection
34

---

Published as conference paper at ICLR 2026
To better understand how context similarity influences the two effects described in Section 2.2, we conducted additional experiments where the context examples are selected using nearest-neighbor similarity rather than random sampling. This allows us probe the extremes of the mechanism behind CoDeC: highly similar context should amplify the helpful-information effect, while random context reflects the dataset's s average similarity structure. We evaluate both seen and unseen datasets under (1) random context (RAND-I), (2) hearest-neighhee context from the same dataset (SIM-1), and (3) nearest-neighbor context from a different dataset with partial distributional overlap. These experiments are not meant to modify CoDeC, but to provide a controlled validation of its underlying intuition and to clarify how similarity and mismatch influence confidence shifts.
As shown in Figure 22a choosing similar samples context reduces the score for unseen MMLU to 0%. However, at the same time it reduces the score for seen datasets as well, leaving the gap mostly unchanged. Clearly, choosing more similar contexts amplifies the positive influence. To preserve practicality of the criterion, the magnitude of that intervention should be carefully chosen to maximize the separation between seen and unseen datasets.
Similarity- based selection has the potential to mitigate adversarial data mixtures. Combining multiple unrelated datasets amplifies the scores due less relevant contexts being sampled, but choosing contexts among the most similar examples can successfully mitigate this issue.
Incorporating KNN-based context sampling introduces additional hyperparameters the pipeline embedding model, similarity metric, how to leverage similarity, etc. Therefore, to preserve the simplicity of the pipeline we decided choose the simplest option of sampling the context uniformly, as sufficient to achieve good results and leaves less decisions for the user. We leave exploring similarity-based context sampling for future work.
35

---

Published as a conference paper at ICLR 2026
D BROADER EVALUATION
Experiments models with known training data, presented in Section confirmed the reliability of CoDeC. With confidence its validity, we subsequently applied CoDeC to broad suite of recent models and analyzed their contamination scores across widely used benchmarks.
Table summarizes the CoDeC scores for various models on popular benchmarks. Most models scored below our high-contaminattion threshold of 80%, suggesting that none were directly trained on benchmark data. However, several scores approach this threshold (e.g.. Phi-4-reasoning-plee on AIME 2024, Devstral-Sman on Reward Bench vl, etc). Such results may indicate partial leakage of benchmark data into training corpora, or extensive training on closely related data (e.g., synthetically generated MMLU questions). In either case, high scores suggest benchmark accuracy may be inflated by memorization rather than genuine reasoning.
AIME 2025, Qwen3-30B 30B-A3B----- --BB-A--Thinning
MATH 500, Brmmorron-Nano88B
A notable case is Qwen 2.5, which scored 100% on the GSM8K training set but only 27% on its set. According to the technical report (Yang et al. (024), GSMSK's training set was deliberately included in training. but questions overlapping heavily with the test were excluded. CoDeC detected this difference, validating its effectiveness. Additionally. it demonstrates that even simple overlap detection strategies can be successful avoid contamination applied carefully.
As discussed earlier, models with outlier scores on given benchmark warrant higher attention. We observe clear examples in IFEval, Hellaswag, and RewardBench vl, where unusually high scores of single models suggest contamination. Hence, accuracy results on such benchmarks should be interpreted with caution for those models.
For GSM8K and Hellaswag, we evaluated both training and test sets. Scores were generally similar across both splits, except for Qwen and related models due to deliberate decontamination. This suggests that other model families, if they directly indirectly use GSM8K training data, made little or no effort to remove overlaps with the test set.
We also computed CoDeC scores for both problems and solutions in the MATH-500 benchmark. Several models scored high on problem statements but all scores on solutions are rather low.
Our evaluation suite includes diverse model types and architectures: base models, instruction-tun tuned models, reasoning optimmzzed models, mixture-ot of----errtttee models, and both older and recent releases. Since CoDeC model-agnostic, it produces meaningful scores regardless of model category. Variants within the same family (e.g., Llamm-3.1--8 vs. I.lama-311 1-88--1-trtuuset typically produce very similar scores, suggesting all variants are equally suitable for CoDeC evaluation.
We observed that the largest models tend to have much lower contamination scores. For instance, Llama-Nemotron 8B appears somewhat contaminated with multiple benchmarks (MMLU-Pro. Reward- Bench v1, AIME 2024, etc.), whereas Llama-Nemotros 235B scores between 0% and 20% across all evaluations. Larger models are known generalize better and thus rely less on memorization. We conjecture that, regardless of overlap between training data and benchmarks, large models depend more on generalization, resulting in lower contamination scores. an analogy, child shown the solution to a quadratic equation may attempt to memorize it, being unable to solve it otherwise; an experienced mathematician, however, would not memorize the example since solving is more straightforward given their knowledge.
36

---

Contamination Detection via Context Broad LLM Evaluation
MATH 500 (problem)
MATH 500 (solution)
LiveCode Bench vl
LiveCodeBench v5
GPQA Diamond
hellaswag (test)
hellaswag (train)
RewardBBench vl
RewardBench v2
AIME 2024
AIME 2025
gamök (test)
gsm8k (trai))
HumanEval
MMLU-PP
BFCL v3
FRAMES
IFEval
BBQ
Model
deepseek Sareeek-ai/Deppeee Di/DeepSoek-RI RI-Distill
Distill-Llama-88 hetpa////eeeeeseet--iaiiil-----
16
20
16
8
7
8
11
20
8
45
12
26
60
7
9
32
7
58
36
7
52
33
deepseekai//DeepSeeeV2
6
0
3
4
19
10
II
2
14
24
19
16
7
33
20
23
EleutherA///yythia-1..t
2
3
0
24
8
28
0
3
3
16
2
7
9
4
12
29
21
36
Fleuther:UUpyyhia-122b
0
3
4
15
25
6
5
II
3
28
2
31
30
5
14
30
19
32
google/gemma--3112bii
3
O
4
17
6
13
11
10
3
2
3
0
155
6
19
17
22
google/gemma-33122b
6
10
O
2
7
24
11
13
1
2
11
O
33
27
5
40
18
29
google/gemmn-3222766
3
O
12
25
5
15
10
11
3
3
2
0
17
6
14
19
31
(oogll//emm---227b-22
3
17
O
3
6
23
12
14
O
II
O
22
18
10
5
39
16
google/gemma-3-41
2
10
12
17
7
16
9
8
3
4
II
5
U
13
18
19
google/
google/gemma-3-4b-p
2
O
4
6
7
32
12
15
2
15
O
44
22
meta- meta-Ilama/L
11
13
0
6
6
19
19
Q
15
17
meta-
Ilama/Llama-331588
8
8
16
8
39
17
15
microsoft/phi-4
75
8
27
19
28
46
37
70
82
12
7
74
29
micrrsott/hii---nniiinntttuuu
52
55
12
39
20
40
58
72
73
53
29
50
microsoft/Phii44rccaoonngg
84
69
12
40
18
36
47
64
81
8
40
41
80
26
55
38
53
Distralai/Deessral-SSall---
60
82
9
7
28
15
2
9
20
17
40
22
33
mistaali/Migiitraa-nalll--333
37
58
8
18
7
28
13
2
28
22
21
32
5
41
26
40
mistralai/Ministral-88
13
27
18
40
13
15
4
16
26
21
45
30
54
mistralai/MMittal--BBB
7
6
8
6
33
8
9
18
0
21
20
13
5
39
25
36
48
58
4
6
31
18
25
3
8
4
34
47
27
50
mistralai/Mixxtal
30
24
4
5
7
26
11
11
2
2
O
15
14
17
7
32
22
nvidia/Llama- Mi-eeooronn-aanee
66
62
37
56
43
10
12
32
nvidia/Llama-3 1aaaama3---Neettrrrrrtnne I-Nemotron--Utra--22222
0
8
12
5
6
12
2
nvidia/
32
12
10
37
49
6
8
9
19
19
Mvidia/MMistrll-eMMMMMMiiii Insstruut
35
34
15
10
25
36
2
7
8
willaa/Vemottroo---77---BB-e
26
O
8
12
38
2
3
4
4
2
24
Ividia/Nemmotron-HHe H-BBB Base-8K
H-566-BBsB-BBB-8
20
10
O
8
7
15
50
2
3
8
30
21
20
Ividia/Vemotrrn----BB-BB.B
38
31
8
19
8
37
13
75
21
6
35
35
49
29
39
Wiidiae/peenRcassnnnnnnNememe 14B
17
20
12
20
20
38
29
30
5
14
16
10
11
34
23
27
widdaa/OOpennReeeoonnnnnneeeeeee
17
13
12
24
21
33
19
27
12
40
14
56
63
40
36
38
WwdiaalQpenRasssnnnnneemoooooo
16
13
25
24
20
37
42
38
4
3
45
22
70
70
38
29
34
60
10
8
25
16
46
99
5
6
60
5
23
13
57
20
49
34
48
Dwenn\ween2.5-155BBBmm
57
27
12
40
17
60
27
100
4
6
43
24
60
16
60
Owen/Qwen2.5-1144
64
10
4
17
13
25
16
70
3
54
13
8
71
12
39
30
Dwen/Qwen2.5-72B
6
4
15
14
45
3
3
48
2
5
6
37
Owen/Owen2.55777
6
8
19
Owen/Qwen3--1BB
7
4
55
17
12
43
12
23
57
96
19
S1
41
35
wwen//Owen33300B-88B-hhhii
40
37
8
39
14
67
Dwen<(Owen3-4B-Thinkii
15
31
16
29
17
28
45
82
6
58
15
31
33
46
13
45
31
50
Owen/Owen3-8B
2
6
8
29
15
30
14
21
5
42
7
8
10
5
35
30
44
Qwen/Qw.Q-32B
7
3
8
36
13
45
3
56
13
12
54
9
48
37
22
5
42
31
59
Table CoDeC scores for widely used models across range of popular benchmarks. Scores are expressed as percentages. Where available, both train and test sets are included in the table.

---

Scores for SPCL v3
120
Scores FRANES
330
Scores forGPGA Damand
-30
Scors for gamde (ima))
for gardik: (tran)
303
Scores for aawag (lasi)
Scores
Humarexal

---

(prasiam)
Scores NATI (stutio)
Scares for VE/LU--m
Scates ResoedBench
110
Score for BesrandZennh 12


Reviews:
[{"review_id": "8RMkD0uUdA", "reviewer": "Reviewer_xj5F", "summary": "The paper introduces CoDeC, a dataset-level method to detect training-data contamination in LLMs by measuring how few-shot, same-dataset context changes model confidence on a target sample. If added context improves confidence, the dataset is likely unseen; if it lowers confidence, the model likely memorized the dataset or closely related data. The per-sample confidence shift Δ(x) is aggregated into a dataset contamination score SCoDeC(D) = (1/N)∑1[Δ(x)<0], requiring only two forward passes per sample and gray-box access to token probabilities. The authors show near-perfect separation of seen vs. unseen datasets (dataset-level AUC ≈ 99.9%) across many models, and provide analyses on training dynamics, finetuning-induced contamination, and robustness.", "questions": "1) Could you provide a small, principled thresholding guide (e.g., validated percentiles on held-out datasets) for users who need binary “contaminated/not” decisions in audits? \n\n2) How does SCoDeC behave under various context sizes n across different dataset lengths/formats? A brief cost–benefit curve would aid deployment. \n\n3) For instruction-tuned/chat-optimized models that yield universally high scores, can lightweight decoding settings (e.g., stop tokens, no-think modes) mitigate anomalies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Clear, elegant idea with strong intuition: leverage in-context learning as a probe—unseen datasets benefit from added context, while memorized datasets get perturbed—yielding a direct, interpretable signal. \n\n2) Practicality & efficiency: model- and dataset-agnostic; only two forward passes per sample; no threshold tuning or access to training corpora needed. \n\n3) Simple, interpretable metric: a percentage score (fraction of samples with negative Δ) that aligns with practitioner intuition and enables straightforward ranking across datasets. \n\n4) Strong empirical evidence: near-perfect dataset-level AUC over diverse models; analyses show early emergence of contamination in training, finetuning-induced contamination, and size trends (larger models memorizing less). \n\n5) Scope beyond strict membership inference: also detects contamination via related/shadow distributions, broadening practical utility for benchmark hygiene.", "weaknesses": "1) Adversarial/degenerate datasets: repeated or highly heterogeneous mixtures can distort the score; the paper notes such edge cases but they remain a limitation for fully automatic use. \n\n2) Calibration to “absolute” labels: SCoDeC is excellent for ranking datasets by contamination risk, but stakeholders may still desire thresholded decisions (ACC/PR/F1). Adding a recommended thresholding recipe could help some users. \n\n3) Model-family anomalies: certain heavily instruction-optimized models behave atypically under CoDeC (e.g., chat/trace behaviors perturb logits broadly), suggesting architecture/task biases may require special handling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0oQGwkmwC", "reviewer": "Reviewer_sLbA", "summary": "This paper introduces Contamination Detection via Context (CoDeC) — a simple, scalable method for detecting and quantifying training data contamination in large language models (LLMs) using in-context learning dynamics.\n\n\nThe key idea\n\n* When given in-context samples from a dataset it has not seen during pre-training, an LLM tends to increase its confidence on the target sample.\n* When the dataset has been seen (i.e., contaminated), additional context disrupts memorization and reduces confidence.\n\nThe method computes a contamination score by measuring the fraction of samples for which adding in-context examples decreases the model’s confidence.\n\n\nMain contributions\n\n* Proposes CoDeC — a model- and dataset-agnostic contamination detection method requiring only gray-box access to logits.\n* Demonstrates near-perfect separation (AUC ≈ 99.9%) between seen and unseen datasets across various models (Pythia, GPT-Neo, RWKV, OLMo, Nemotron).\n* Shows robustness to dataset diversity and training stage, and scalability to large benchmarks.\n* Highlights how contamination transfers across related datasets and how finetuning affects CoDeC scores.\n* Positions CoDeC as an interpretable, efficient alternative to classical membership inference methods.", "questions": "Empirical clarification\n\n* Can the authors provide empirical evidence or references to support the claims in Section 2.2 and Section 3.5 regarding confidence shifts and generalization ability?\n* Can they quantify how CoDeC correlates with generalization metrics across models with similar benchmark accuracy?\n\nContamination vs distributional similarity\n\n* How robust is CoDeC to partial contamination or to unseen but stylistically similar datasets?\n* Could the authors provide additional results where datasets are synthetically perturbed to vary similarity to training data?\n\nThresholding strategy\n\n* The paper mentions thresholding and comparison against other models. Could the authors elaborate on practical guidelines for selecting these thresholds in real-world evaluation scenarios?\n\nTheoretical properties\n\n* Do the authors have any theoretical guarantees (e.g., consistency or error bounds) on CoDeC scores? Or atleast outline what such analysis might require?\n\nReproducibility and usage\n\n* How sensitive is the method to the number of in-context samples and the randomness of context selection?\n* Would deterministic context selection (e.g., nearest neighbors) further improve stability?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality\n\n* While prior work relies heavily on membership inference, loss-based calibration, or reference models, CoDeC offers a novel formulation by exploiting in-context learning behavior as a contamination signal.\n* The idea is conceptually elegant — turning a standard model property (ICL behavior) into a contamination test.\n\nQuality\n\n* The experiments are extensive: multiple models (spanning architectures and sizes), datasets (training vs unseen), and baselines.\n* Ablation studies are included (context size, dataset size) and finetuning experiments provide additional validation.\n* Clear definition of the contamination score and rationale for why it works.\n\nClarity\n\n* The paper is very well written, with clean exposition of the problem statement and intuition.\n* Figures are informative and minimalistic, making the method easy to grasp.\n* The pipeline (Fig. 1) clearly communicates the method steps.\n\nSignificance\n\n* Data contamination detection is increasingly critical for fair evaluation of LLMs.\n* A method that is scalable, interpretable, and does not require access to training data or reference models is very valuable.\n* The simplicity and efficiency of CoDeC make it applicable at scale, which is practically important.", "weaknesses": "Experiment rigor to test the generality of the findings\n\n* The paper claims with the help of CMA that LLMs \n* Similar unsubstantiated claims appear in Section 2.2 (Key Idea), e.g., that adding in-context samples usually improves confidence for unseen datasets — no citations or empirical backing are provided there.\n\nLimited theoretical grounding\n\n* While intuition is discussed in depth (Section 2.5), the theoretical explanation remains qualitative.\n* Formal guarantees or bounds on false positive/negative rates for contamination detection would strengthen the contribution.\n\nEvaluation scope\n\n* Although contamination transfer across related datasets (e.g., MMLU) is explored, adversarial or edge cases (e.g., near-identical but unseen datasets, noisy mixtures) are only lightly discussed.\n* The method might conflate contamination with related distribution overlap, as the authors acknowledge, but do not quantitatively evaluate how severe this is.\n\nNovelty relative to related work\n\n* Some elements overlap conceptually with loss-based or entropy-based detection. Positioning CoDeC more clearly in terms of unique advantages and trade-offs would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0FvwmZEKT", "reviewer": "Reviewer_7wL6", "summary": "The authors propose a method called CoDeC (Contamination Detection via Context) for detecting dataset-level contamination. The key idea is simple yet effective, for unseen data ICL examples will boost its confidence, so Delta (=ICL - no ICL) would be positive, but for trained data the ICL examples could be confusing hence result in negative Delta. The authors conducted wide range of experiments to support their method.", "questions": "1. Does model's capability have a correlation with the method effectiveness? Could you also report the general eval metrics (for benchmarks other that the contaminated data) of the models before and after finetuning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clean and simple idea. I always appreciate simple ideas that work, and this paper's idea is novel and effective. \n2. Extensive experiments. The authors conducted experiments on a wide range of model families (even for RWKV ones), all with strong experiment results.\n3. Well-written and of importance to the field. The writing is clear and the topic is important to the community.", "weaknesses": "I have 2 key questions:\n1. Does CoDeC work when you rephrase the question? (either train on rephrased versions then test on original, or train on original then test on the rephrased) I think it's getting widely known that many contamination cannot detect contamination unless it's under exact same phrasing, and arguably the value is low if a method cannot work for general use cases.\n2. Did you try training with a mix of other datasets in your experiments? i.e. not only the contamination test set, but with general corpus? This might improve models general ICL ability and render CoDeC ineffective, and people usually train their model this way instead of only using the test set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)